peterxcli opened a new pull request, #58239: URL: https://github.com/apache/spark/pull/58239
### What changes were proposed in this pull request? This PR fixes [SPARK-58949](https://issues.apache.org/jira/browse/SPARK-58949) by: - sorting newly written Variant object fields by unsigned lexicographic UTF-8 bytes, as required by the [Variant encoding specification](https://github.com/apache/parquet-format/blob/24102ed5c56e51b610a4897e5f79e76e43732d1d/VariantEncoding.md#L449-L463); - using binary search for object lookup at every object size and comparing query bytes directly with the metadata dictionary; - retrying lookup with Java UTF-16 order when needed so values written by older Spark versions remain readable; and - accepting both canonical UTF-8 order and legacy UTF-16 order during schema validation, while preserving the established schema field order. This follows the compatibility direction discussed in [apache/parquet-java#3736](https://github.com/apache/parquet-java/pull/3736). ### Why are the changes needed? The Variant specification orders object keys by unsigned UTF-8 bytes, but Spark used String.compareTo, which orders UTF-16 code units. These orders differ for some valid keys. For example, UTF-16 places U+10000 before U+FFFF, while unsigned UTF-8 places U+FFFF first. As a result, Spark wrote non-canonical Variant objects and could miss fields when binary-searching canonical values produced by another implementation. ### Does this PR introduce _any_ user-facing change? Yes. Newly written Variant objects use the specification's unsigned UTF-8 field order. Spark continues to read affected values written in the legacy UTF-16 order, and schema output keeps its existing field order. ### How was this patch tested? Added regressions for canonical and legacy object lookup, nested objects, schema_of_variant, and Parquet shredding-schema inference: build/sbt \ 'catalyst/testOnly *VariantExpressionSuite -- -z "SPARK-58949"' \ 'sql/testOnly *VariantInferShreddingSuite -- -z "SPARK-58949"' Both suites passed (1 test each). The affected modules also passed Java checkstyle and main/test scalastyle. I also ran a temporary lookup microbenchmark on an Apple M4 with Zulu OpenJDK 21.0.6, comparing upstream/master at 9da9f8d6739 with this patch at dda8e446f00. Each result is the median of three alternating JVM fork medians. Each fork used a fixed 2 GiB heap, 5 seconds of warmup per case, and 9 measured rounds of 5,000,000 lookups. Object construction was excluded. Lower is better. | Object / lookup | master (ns/op) | patch (ns/op) | Change | | --- | ---: | ---: | ---: | | 16 fields, ASCII present | 195.6 | 56.6 | -71.1% | | 16 fields, ASCII absent | 193.8 | 49.4 | -74.5% | | 256 fields, canonical present | 100.2 | 78.0 | -22.2% | | 256 fields, canonical absent | 113.1 | 72.5 | -35.9% | | 256 fields, legacy fallback present | 175.1 | 217.3 | +24.1% | | 256 fields, legacy fallback absent | 215.1 | 279.0 | +29.7% (approx.) | The correctness assertion for a canonical U+10000 lookup changes from false on master to true with this patch. The extra compatibility search costs about 24-30% for the measured legacy/divergent-Unicode cases; ordinary ASCII lookups remain single-pass and improve by 22-75%. The legacy-absent timing was noisier across fork pairs, so its aggregate is marked approximate. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
