peterxcli opened a new issue, #5474: URL: https://github.com/apache/datafusion-comet/issues/5474
### What is the problem the feature request solves? PR #5407 must bridge two incompatible Variant object-key orders: - Parquet Variant and Arrow use unsigned lexicographic UTF-8 byte order. - Supported Spark 4.x versions build and search Variant objects using Java UTF-16 order, and existing Spark-written Parquet files can therefore contain that legacy order. Comet currently preserves correctness by recursively examining and sometimes re-encoding Variant values at the native Parquet normalization boundary: - [Normalized output is converted to Spark UTF-16 order](https://github.com/apache/datafusion-comet/blob/ba37a688e5aaa1d9e4a17c55d42ea0211486f7a3/native/core/src/parquet/cast_column/variant.rs#L38-L87). - [`SparkMetadataBuilder` and the recursive compatibility scan implement both orderings](https://github.com/apache/datafusion-comet/blob/ba37a688e5aaa1d9e4a17c55d42ea0211486f7a3/native/core/src/parquet/cast_column/variant.rs#L394-L478). - [Partially shredded residual values are temporarily converted to Arrow UTF-8 order before unshredding](https://github.com/apache/datafusion-comet/blob/ba37a688e5aaa1d9e4a17c55d42ea0211486f7a3/native/core/src/parquet/cast_column/variant.rs#L190-L225). - [Shredded values are rebuilt for Spark compatibility](https://github.com/apache/datafusion-comet/blob/ba37a688e5aaa1d9e4a17c55d42ea0211486f7a3/native/core/src/parquet/cast_column/variant.rs#L967-L1048). This compatibility layer is necessary today, but it adds recursive per-row traversal, allocations when re-encoding is required, and substantial code in the scan path. This is a focused follow-up to #5438 and #5407. ### Describe the potential solution After every supported Spark 4 profile writes canonical UTF-8-ordered Variant objects and reads both canonical UTF-8 and legacy UTF-16 data: 1. Export Arrow-rs canonical Variant bytes directly instead of converting normalized output to Spark UTF-16 order. 2. Remove `SparkMetadataBuilder`, `VariantObjectKeyOrder::SparkUtf16`, and the recursive output compatibility/re-encoding path. 3. Preserve only the narrow input-side conversion needed for legacy Spark Parquet files whose residual `value` is UTF-16 ordered. Prefer upstream Arrow-rs support; otherwise document when that compatibility path can be retired. 4. Benchmark whole-value Variant projection before and after the change so the removed traversal has measured impact. Definition of done: - Canonical Parquet Variant files and legacy Spark-written files remain readable. - Top-level, nested, and partially shredded objects with Unicode keys round-trip correctly. - Spark field lookup works for objects above its binary-search threshold. - Native output no longer performs a recursive UTF-16 rewrite. - Obsolete compatibility code and tests are removed, or reduced to a documented legacy-input path. - Spark 3.x behavior remains unchanged. ### Additional context - [SPARK-58949](https://issues.apache.org/jira/browse/SPARK-58949) tracks the Spark ordering incompatibility. - [Spark PR #58239](https://github.com/apache/spark/pull/58239) proposed canonical UTF-8 writes/search with legacy UTF-16 read fallback, but closed unmerged; Spark still needs a landed replacement. - [parquet-java issue #3735](https://github.com/apache/parquet-java/issues/3735) tracks the same ordering problem. - [parquet-java PR #3746](https://github.com/apache/parquet-java/pull/3746) is open and implements canonical UTF-8 write/search plus legacy UTF-16 lookup fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
