I filed a ticket to track this work[1] and also perhaps to gather some additional help / collaboration.
[1]: https://github.com/apache/parquet-testing/issues/75 On Mon, Apr 7, 2025 at 6:02 AM Andrew Lamb <andrewlam...@gmail.com> wrote: > Thank you very much David > > I will try to create some examples this week and report back. > > Andrew > > On Sun, Apr 6, 2025 at 4:48 PM David Cashman > <david.cash...@databricks.com.invalid> wrote: > >> Hi Andrew, you should be able to create shredded files using OSS Spark >> 4.0. I think the only issue is that it doesn't have the logical type >> annotation yet, so readers wouldn't be able to distinguish it from a >> non-variant struct that happens to have the same schema. (Spark is >> able to infer that it is a Variant from the >> `org.apache.spark.sql.parquet.row.metadata` metadata.) >> >> The ParquetVariantShreddingSuite in Spark has some tests that write >> and read shredded parquet files. Below is an example that translates >> the first test into code that runs in spark-shell and writes a Parquet >> file. The shredding schema is set via conf. If you want to test types >> that Spark doesn't infer in parse_json (e.g. timestamp, binary), you >> can use `to_variant_object` to cast structured values to Variant. >> >> I won't have time to work on this in the next couple of weeks, but am >> happy to answer any questions. >> >> Thanks, >> >> David >> >> scala> import org.apache.spark.sql.internal.SQLConf >> scala> spark.conf.set(SQLConf.VARIANT_WRITE_SHREDDING_ENABLED.key, true) >> scala> spark.conf.set(SQLConf.VARIANT_ALLOW_READING_SHREDDED.key, true) >> scala> spark.conf.set(SQLConf.VARIANT_FORCE_SHREDDING_SCHEMA_FOR_TEST.key, >> "a int, b string, c decimal(15, 1)") >> scala> val df = spark.sql( >> | """ >> | | select case >> | | when id = 0 then parse_json('{"a": 1, "b": "2", "c": >> 3.3, "d": 4.4}') >> | | when id = 1 then parse_json('{"a": [1,2,3], "b": >> "hello", "c": {"x": 0}}') >> | | when id = 2 then parse_json('{"A": 1, "c": 1.23}') >> | | end v from range(3) >> | |""".stripMargin) >> scala> df.write.mode("overwrite").parquet("/tmp/shredded_test") >> scala> spark.read.parquet("/tmp/shredded_test").show >> +--------------------+ >> | v| >> +--------------------+ >> |{"a":1,"b":"2","c...| >> |{"a":[1,2,3],"b":...| >> | {"A":1,"c":1.23}| >> +--------------------+ >> >> >> On Sun, Apr 6, 2025 at 2:54 PM Andrew Lamb <andrewlam...@gmail.com> >> wrote: >> > >> > Can someone (pretty pretty) please give us some binary examples so we >> can >> > make faster progress on the Rust implementation? >> > >> > We recently got exciting news[1] that folks from the CMU database group >> > have started working on the Rust implementation of variant, and I would >> > very much like to encourage and support their work. >> > >> > I am willing to do some legwork (make a PR to parquet-testing for >> example) >> > if someone can point me to the files (or instructions on how to use some >> > system to create variants). >> > >> > I was hoping that since the VARIANT format[2] and draft shredding >> spec[3] >> > have been in the repo for 6 months (since October 2024) , it would be >> > straightforward to provide some examples. Do we know anything that is >> > blocking the creation of examples? >> > >> > Andrew >> > >> > [1]: >> https://github.com/apache/arrow-rs/issues/6736#issuecomment-2781556103 >> > [2]: >> https://github.com/apache/parquet-format/blob/master/VariantEncoding.md >> > [3]: >> > >> https://github.com/apache/parquet-format/blob/master/VariantShredding.md >> > >> > >> > On Wed, Mar 5, 2025 at 3:58 PM Julien Le Dem <jul...@apache.org> wrote: >> > >> > > That sounds like a great suggestion to me. >> > > >> > > On Wed, Mar 5, 2025 at 12:41 PM Andrew Lamb <andrewlam...@gmail.com> >> > > wrote: >> > > >> > > > I would like to request before the VARIANT spec changes are >> finalized >> > > that >> > > > we have example data in parquet-testing. >> > > > >> > > > This topic came up (well, I brought it up) on the sync call today. >> > > > >> > > > In my opinion, having example files would reduce the overhead of new >> > > > implementations dramatically. At least there should be example of >> > > > * variant columns (no shredding) >> > > > * variant columns with shredding >> > > > >> > > > Some description of what those files contained ("expected >> contents"). For >> > > > prior art, here is what Dewey did for the geometry type[1][2]. >> > > > >> > > > When looking for prior discussions, I found a great quote from Gang >> Wu[3] >> > > > on this topic: >> > > > >> > > > > I'd say that a lesson learned is that we should publish example >> files >> > > > for any >> > > > > new feature to the parquet-testing [1] repo for interoperability >> tests. >> > > > >> > > > Thank you for your consideration, >> > > > Andrew >> > > > >> > > > >> > > > >> > > > >> > > > [1] https://github.com/apache/parquet-testing/pull/70 >> > > > [2] https://github.com/geoarrow/geoarrow-data >> > > > [3]: >> https://lists.apache.org/thread/71d7p9lprhf514jnt5dgnw4wfmn8ykzt >> > > > >> > > >> >
