Hey Gang, Thanks for raising this. +1 from my end.
For context, as Gang mentioned, when proposing to add a Variant Type to Iceberg <https://github.com/apache/iceberg/issues/10392>, one of the future goals was to integrate more closely with Parquet, and having the spec at Parquet will help to speed this up. Kind regards, Fokko Op vr 23 aug 2024 om 11:37 schreef Gábor Szádovszky <[email protected]>: > Hi Gang, > > Thanks for bringing this up. > > I think that if Variant type would have come up earlier (before > iceberg/arrow), its natural place would have been at the file format level > as any other types. The communities started discussing where it should be > placed because now we have different type systems at different places. > Also, the current spec of Variant makes it more or less independent from > the Parquet file format. > However, even at Parquet level, we would need at least an additional > Logical type to help handle Variant type by the systems reading/writing > Parquet. > > To summarize my opinion, +1 for having the whole Variant spec in Parquet > format. > > Cheers, > Gabor > > Gang Wu <[email protected]> ezt írta (időpont: 2024. aug. 23., P, 11:18): > > > Hi, > > > > Apache Iceberg is adding variant type support [1][2] by adopting the > > variant > > spec [3] from Apache Spark. As the proposal is getting mature, both > Iceberg > > [4] > > and Spark [5] communities are discussing moving the variant type to > Parquet > > repo to avoid divergence. Moving it into Parquet makes the variant spec > > engine > > and table format agnostic, which may encourage wider adoption. > > > > What do people from Parquet community think? > > > > [1] https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34 > > [2] https://lists.apache.org/thread/xcyytoypgplfr74klg1z2rgjo6k5b0sq > > [3] > > > > > https://github.com/apache/spark/blob/d84f1a3575c4125009374521d2f179089ebd71ad/common/variant/README.md > > [4] https://lists.apache.org/thread/hopkr2f0ftoywwt9zo3jxb7n0ob5s5bw > > [5] https://lists.apache.org/thread/0k5oj3mn0049fcxoxm3gx3d7r28gw4rj > > > > Best, > > Gang > > >
