pan3793 commented on PR #49874: URL: https://github.com/apache/spark/pull/49874#issuecomment-2653036367
@gene-db @cloud-fan I read the following docs to learn the new added variant feature, and haven't deep dive the code. - https://github.com/apache/spark/blob/master/common/variant/README.md - https://github.com/apache/spark/blob/master/common/variant/shredding.md I didn't realize that Spark implemented the updated https://github.com/apache/parquet-format/pull/461 before it get merged, I see there are some discussions happen in 2025/01, and some unmerged PRs related to Variant spec, e.g. https://github.com/apache/parquet-format/pull/474 And I also see statements like below in Spark's Variant PRs, e.g. #49234 > ... is currently disabled, and only usable as a test feature. Given the comments from @gene-db, I think that the Spark's variant docs are outdated? If so, could you please update them, or simply link to Parquet's repo? And could you please clarify the status of Variant feature in Spark 4.0, is _"... is currently disabled, and only usable as a test feature"_ still true? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
