Thanks Aihua, We are not (yet) blocked on development by the logical type annotation, though I am blocked from creating example parquet files
Do you think it would be possible to create annotated parquet files using arrow-cpp now that Neil's PR[1] has been merged? I haven't really tried yet Andrew [1]: https://github.com/apache/arrow/pull/45375 On Thu, May 8, 2025 at 9:59 PM Aihua Xu <aihu...@gmail.com> wrote: > Since we haven’t released any parquet version with variant logical type, I > don’t think any engine would write that yet. > > Are you getting blocked by this? Can you workaround the issue for now? > > Thanks > Aihua > > > > On May 8, 2025, at 12:57 PM, Andrew Lamb <andrewlam...@gmail.com> wrote: > > > > Update here: the initial example files[1] were merged (thanks Micah!) > > > > However, Spark 4.0 (and what is on the main branch) does not appear to > > write parquet files with the Logical annotations yet (it uses its own > > metadata it seems) -- you can see what I tried in [2]. > > > > **Does anyone know of a system that can write Variant values with the > > proper Parquet logical type?** > > > > Thanks, > > Andrew > > > > > > > > [1]: https://github.com/apache/parquet-testing/pull/76 > > [2]: > > > https://github.com/apache/parquet-testing/issues/75#issuecomment-2847815424 > > > > > > > >> On Fri, May 2, 2025 at 2:22 PM Andrew Lamb <andrewlam...@gmail.com> > wrote: > >> > >> Thanks to Micah, I think we have the first PR with example Variant > >> values[1] almost ready to merge. > >> > >> Next up will be figuring out how to create Parquet files with the proper > >> logical annotations. > >> > >> Andrew > >> > >> p.s. In case it isn't obvious I would like introducing Variant into > >> Parquet to be a model of how to extend the spec and get wide adoption > >> across the ecosystem quickly, for two reasons: > >> 1. the actual Variant funtionality > >> 2. To counteract the narrative that Parquet is ossified and not possible > >> to change. > >> > >> I personally think adding the binary examples is critical to > >> helping other language implementations. > >> > >> [1]: https://github.com/apache/parquet-testing/pull/76 > >> > >> On Wed, Apr 16, 2025 at 10:59 AM Andrew Lamb <andrewlam...@gmail.com> > >> wrote: > >> > >>> Update here is I have created a PR with example variant values (not yet > >>> parquet files, just the variant values)[1]. > >>> > >>> Since Spark seems to be the only open source software capable of > creating > >>> variants at this time, I generated the examples using Spark. > >>> > >>> Please check it out and let me know what you think. If it is > acceptable I > >>> can work on PRs (based on Aihua's example) for actual parquet files > with > >>> encoded values > >>> > >>> Andrew > >>> > >>> [1]: https://github.com/apache/parquet-testing/pull/76 > >>> > >>> On Tue, Apr 8, 2025 at 5:46 AM Andrew Lamb <andrewlam...@gmail.com> > >>> wrote: > >>> > >>>> Thank you very much > >>>> > >>>> On Mon, Apr 7, 2025 at 5:10 PM Aihua Xu <aihu...@gmail.com> wrote: > >>>> > >>>>> I attached in the > https://github.com/apache/parquet-testing/issues/75. > >>>>> Please rename them back to *.parquet so you can use parquet tools to > >>>>> view > >>>>> them. > >>>>> > >>>>> I captured them when working on the Iceberg tests in > >>>>> > >>>>> > https://github.com/apache/iceberg/blob/main/parquet/src/test/java/org/apache/iceberg/parquet/TestVariantWriters.java#L172 > >>>>> . > >>>>> > >>>>> You can change to OutputFile outputFile = > >>>>> Files.localOutput("primitive.parquet"); to capture them, but you > >>>>> probably > >>>>> can follow what David mentioned. > >>>>> > >>>>> On Mon, Apr 7, 2025 at 12:26 PM Andrew Lamb <andrewlam...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Attaching them on the ticket[1] would also be a way to share them > >>>>>> > >>>>>> It would also be super helpful to share the commands you ran > >>>>>> > >>>>>> Andrew > >>>>>> > >>>>>> [1]: https://github.com/apache/parquet-testing/issues/75 > >>>>>> > >>>>>> On Mon, Apr 7, 2025 at 2:46 PM Adrian Garcia Badaracco > >>>>>> <adr...@pydantic.dev.invalid> wrote: > >>>>>> > >>>>>>> Yes I am not able to see them. Could you make a PR to the repo, or > >>>>> upload > >>>>>>> them somewhere so we can make a PR? Even if it doesn’t get merged > >>>>>>> immediately we can pull them from the PR. Thanks! > >>>>>>> > >>>>>>> On Mon, Apr 7, 2025 at 1:44 PM Aihua Xu <aihu...@gmail.com> wrote: > >>>>>>> > >>>>>>>> Hi Adrian, > >>>>>>>> > >>>>>>>> I attached them to my reply and I'm not sure if the files get > >>>>> filtered. > >>>>>>> Let > >>>>>>>> me know if you still can't see them. Maybe I should push to the > >>>>> repo > >>>>>>>> instead. > >>>>>>>> > >>>>>>>> On Mon, Apr 7, 2025 at 10:58 AM Adrian Garcia Badaracco > >>>>>>>> <adr...@pydantic.dev.invalid> wrote: > >>>>>>>> > >>>>>>>>> Amazing Aihua, thanks so much! > >>>>>>>>> > >>>>>>>>> Sorry if I just missed it but... where are the files you > >>>>> created? I > >>>>>>> don't > >>>>>>>>> see them in the repo / issue / this thread. > >>>>>>>>> > >>>>>>>>> On Mon, Apr 7, 2025 at 12:54 PM Aihua Xu <aihu...@gmail.com> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>>> I have created some test files attached during the > >>>>> development, one > >>>>>>> is > >>>>>>>>>> without shredding and one is with shredding. > >>>>>>>>>> > >>>>>>>>>> As David pointed out, it's missing the Variant logical type > >>>>> but > >>>>>> you > >>>>>>>> can > >>>>>>>>>> use that as reference and as a start. > >>>>>>>>>> > >>>>>>>>>> On Mon, Apr 7, 2025 at 3:50 AM Andrew Lamb < > >>>>> andrewlam...@gmail.com > >>>>>>> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> I filed a ticket to track this work[1] and also perhaps to > >>>>> gather > >>>>>>> some > >>>>>>>>>>> additional help / collaboration. > >>>>>>>>>>> > >>>>>>>>>>> [1]: https://github.com/apache/parquet-testing/issues/75 > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Apr 7, 2025 at 6:02 AM Andrew Lamb < > >>>>>> andrewlam...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Thank you very much David > >>>>>>>>>>>> > >>>>>>>>>>>> I will try to create some examples this week and report > >>>>> back. > >>>>>>>>>>>> > >>>>>>>>>>>> Andrew > >>>>>>>>>>>> > >>>>>>>>>>>> On Sun, Apr 6, 2025 at 4:48 PM David Cashman > >>>>>>>>>>>> <david.cash...@databricks.com.invalid> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Andrew, you should be able to create shredded files > >>>>> using > >>>>>> OSS > >>>>>>>>> Spark > >>>>>>>>>>>>> 4.0. I think the only issue is that it doesn't have the > >>>>> logical > >>>>>>>> type > >>>>>>>>>>>>> annotation yet, so readers wouldn't be able to > >>>>> distinguish it > >>>>>>> from > >>>>>>>> a > >>>>>>>>>>>>> non-variant struct that happens to have the same schema. > >>>>> (Spark > >>>>>>> is > >>>>>>>>>>>>> able to infer that it is a Variant from the > >>>>>>>>>>>>> `org.apache.spark.sql.parquet.row.metadata` metadata.) > >>>>>>>>>>>>> > >>>>>>>>>>>>> The ParquetVariantShreddingSuite in Spark has some tests > >>>>> that > >>>>>>> write > >>>>>>>>>>>>> and read shredded parquet files. Below is an example that > >>>>>>>> translates > >>>>>>>>>>>>> the first test into code that runs in spark-shell and > >>>>> writes a > >>>>>>>>> Parquet > >>>>>>>>>>>>> file. The shredding schema is set via conf. If you want > >>>>> to test > >>>>>>>> types > >>>>>>>>>>>>> that Spark doesn't infer in parse_json (e.g. timestamp, > >>>>>> binary), > >>>>>>>> you > >>>>>>>>>>>>> can use `to_variant_object` to cast structured values to > >>>>>> Variant. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I won't have time to work on this in the next couple of > >>>>> weeks, > >>>>>>> but > >>>>>>>> am > >>>>>>>>>>>>> happy to answer any questions. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> > >>>>>>>>>>>>> David > >>>>>>>>>>>>> > >>>>>>>>>>>>> scala> import org.apache.spark.sql.internal.SQLConf > >>>>>>>>>>>>> scala> > >>>>>>> spark.conf.set(SQLConf.VARIANT_WRITE_SHREDDING_ENABLED.key, > >>>>>>>>>>> true) > >>>>>>>>>>>>> scala> > >>>>>> spark.conf.set(SQLConf.VARIANT_ALLOW_READING_SHREDDED.key, > >>>>>>>>> true) > >>>>>>>>>>>>> scala> > >>>>>>>>>>> > >>>>>> spark.conf.set(SQLConf.VARIANT_FORCE_SHREDDING_SCHEMA_FOR_TEST.key, > >>>>>>>>>>>>> "a int, b string, c decimal(15, 1)") > >>>>>>>>>>>>> scala> val df = spark.sql( > >>>>>>>>>>>>> | """ > >>>>>>>>>>>>> | | select case > >>>>>>>>>>>>> | | when id = 0 then parse_json('{"a": 1, > >>>>> "b": > >>>>>> "2", > >>>>>>>> "c": > >>>>>>>>>>>>> 3.3, "d": 4.4}') > >>>>>>>>>>>>> | | when id = 1 then parse_json('{"a": > >>>>> [1,2,3], > >>>>>> "b": > >>>>>>>>>>>>> "hello", "c": {"x": 0}}') > >>>>>>>>>>>>> | | when id = 2 then parse_json('{"A": 1, > >>>>> "c": > >>>>>>> 1.23}') > >>>>>>>>>>>>> | | end v from range(3) > >>>>>>>>>>>>> | |""".stripMargin) > >>>>>>>>>>>>> scala> > >>>>> df.write.mode("overwrite").parquet("/tmp/shredded_test") > >>>>>>>>>>>>> scala> spark.read.parquet("/tmp/shredded_test").show > >>>>>>>>>>>>> +--------------------+ > >>>>>>>>>>>>> | v| > >>>>>>>>>>>>> +--------------------+ > >>>>>>>>>>>>> |{"a":1,"b":"2","c...| > >>>>>>>>>>>>> |{"a":[1,2,3],"b":...| > >>>>>>>>>>>>> | {"A":1,"c":1.23}| > >>>>>>>>>>>>> +--------------------+ > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sun, Apr 6, 2025 at 2:54 PM Andrew Lamb < > >>>>>>> andrewlam...@gmail.com > >>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Can someone (pretty pretty) please give us some binary > >>>>>> examples > >>>>>>>> so > >>>>>>>>> we > >>>>>>>>>>>>> can > >>>>>>>>>>>>>> make faster progress on the Rust implementation? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We recently got exciting news[1] that folks from the CMU > >>>>>>> database > >>>>>>>>>>> group > >>>>>>>>>>>>>> have started working on the Rust implementation of > >>>>> variant, > >>>>>>> and I > >>>>>>>>>>> would > >>>>>>>>>>>>>> very much like to encourage and support their work. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I am willing to do some legwork (make a PR to > >>>>> parquet-testing > >>>>>>> for > >>>>>>>>>>>>> example) > >>>>>>>>>>>>>> if someone can point me to the files (or instructions > >>>>> on how > >>>>>> to > >>>>>>>> use > >>>>>>>>>>> some > >>>>>>>>>>>>>> system to create variants). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I was hoping that since the VARIANT format[2] and draft > >>>>>>> shredding > >>>>>>>>>>>>> spec[3] > >>>>>>>>>>>>>> have been in the repo for 6 months (since October 2024) > >>>>> , it > >>>>>>>> would > >>>>>>>>> be > >>>>>>>>>>>>>> straightforward to provide some examples. Do we know > >>>>> anything > >>>>>>>> that > >>>>>>>>> is > >>>>>>>>>>>>>> blocking the creation of examples? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1]: > >>>>>>>>>>>>> > >>>>>>>>> > >>>>>> > >>>>> > https://github.com/apache/arrow-rs/issues/6736#issuecomment-2781556103 > >>>>>>>>>>>>>> [2]: > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> > >>>>>> > >>>>> > https://github.com/apache/parquet-format/blob/master/VariantEncoding.md > >>>>>>>>>>>>>> [3]: > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>> > https://github.com/apache/parquet-format/blob/master/VariantShredding.md > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Wed, Mar 5, 2025 at 3:58 PM Julien Le Dem < > >>>>>>> jul...@apache.org> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> That sounds like a great suggestion to me. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Wed, Mar 5, 2025 at 12:41 PM Andrew Lamb < > >>>>>>>>>>> andrewlam...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I would like to request before the VARIANT spec > >>>>> changes > >>>>>> are > >>>>>>>>>>>>> finalized > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> we have example data in parquet-testing. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This topic came up (well, I brought it up) on the > >>>>> sync > >>>>>> call > >>>>>>>>>>> today. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> In my opinion, having example files would reduce the > >>>>>>> overhead > >>>>>>>>> of > >>>>>>>>>>> new > >>>>>>>>>>>>>>>> implementations dramatically. At least there should > >>>>> be > >>>>>>>> example > >>>>>>>>> of > >>>>>>>>>>>>>>>> * variant columns (no shredding) > >>>>>>>>>>>>>>>> * variant columns with shredding > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Some description of what those files contained > >>>>> ("expected > >>>>>>>>>>>>> contents"). For > >>>>>>>>>>>>>>>> prior art, here is what Dewey did for the geometry > >>>>>>>> type[1][2]. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> When looking for prior discussions, I found a great > >>>>> quote > >>>>>>>> from > >>>>>>>>>>> Gang > >>>>>>>>>>>>> Wu[3] > >>>>>>>>>>>>>>>> on this topic: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I'd say that a lesson learned is that we should > >>>>>> publish > >>>>>>>>>>> example > >>>>>>>>>>>>> files > >>>>>>>>>>>>>>>> for any > >>>>>>>>>>>>>>>>> new feature to the parquet-testing [1] repo for > >>>>>>>>>>> interoperability > >>>>>>>>>>>>> tests. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thank you for your consideration, > >>>>>>>>>>>>>>>> Andrew > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> [1] > >>>>> https://github.com/apache/parquet-testing/pull/70 > >>>>>>>>>>>>>>>> [2] https://github.com/geoarrow/geoarrow-data > >>>>>>>>>>>>>>>> [3]: > >>>>>>>>>>>>> > >>>>>> https://lists.apache.org/thread/71d7p9lprhf514jnt5dgnw4wfmn8ykzt > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> >
