peterxcli opened a new pull request, #3746: URL: https://github.com/apache/parquet-java/pull/3746
### Rationale for this change The [Variant encoding specification](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) requires the field ids in an object's header to be sorted by the unsigned byte order of the field names' UTF-8 encoding, so readers can binary-search them. `VariantBuilder` sorted the fields — and `Variant.getFieldByKey` binary-searched them — using `String.compareTo`, which orders UTF-16 code units instead. The two orderings agree for all keys in the Basic Multilingual Plane but diverge for supplementary-plane characters (U+10000 and above): `String.compareTo` orders a leading high surrogate (0xD800–0xDBFF) before code points in U+E000..U+FFFF, whereas UTF-8 byte order (and the spec) orders them after. Consequences: - Objects parquet-java builds with such keys have field ids sorted in a spec-violating order, so spec-compliant readers can fail to find fields via binary search. - parquet-java's own binary search can fail to find a supplementary-plane key in an object produced by a spec-compliant writer. This adapts #3736 by @rayokota and adds the read-compatibility fallback from the equivalent Spark fix ([apache/spark#58239](https://github.com/apache/spark/pull/58239)), per the discussion on that PR. ### What changes are included in this PR? - New `VariantUtil.encodeKey(String)` and `VariantUtil.compareKeys(byte[], byte[])`, which order field names by unsigned lexicographic UTF-8 byte order (from #3736) - `VariantBuilder.FieldEntry.compareTo` sorts object fields with that comparison, lazily caching each field's UTF-8 encoding (from #3736) - `Variant.getFieldByKey` binary-searches in UTF-8 byte order first; for keys containing a code unit at or above U+D800 (the only keys where the two orders can differ), it retries the search in UTF-16 order, so objects written by older versions in the legacy order remain readable ### Are these changes tested? Three new tests in `TestVariantObjectBuilder`: - `testObjectKeysSortedByUtf8ByteOrder` — builds an object with keys U+FFFF (`EF BF BF`) and U+10000 (`F0 90 80 80`) appended in reverse and asserts the encoded field order is U+FFFF then U+10000 (UTF-8 order), which the previous `compareTo` reversed. - `testLargeObjectBinarySearchWithSupplementaryKey` — a 42-field object (above `BINARY_SEARCH_THRESHOLD`) mixing ASCII keys with U+FFFF and U+10000, asserting `getFieldByKey` resolves both through the binary-search path. - `testLegacyUtf16OrderedObjectLookup` — rewrites a canonical object's id and offset lists into the legacy UTF-16 order and asserts `getFieldByKey` still finds ASCII, U+FFFF, and U+10000 keys through the fallback search, and that absent keys stay absent. All 183 `parquet-variant` tests and the `parquet-avro` variant read/write suites pass locally. ### Are there any user-facing changes? Newly written Variant objects containing supplementary-plane field keys now use the specification's unsigned UTF-8 field order. Objects written in the legacy UTF-16 order remain readable via `getFieldByKey`. Closes #3735 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
