peterxcli opened a new pull request, #126: URL: https://github.com/apache/parquet-testing/pull/126
### Rationale The [Variant spec] requires object fields to be ordered by **unsigned lexicographic UTF-8 bytes** of their names. For supplementary characters this differs from UTF-16 code-unit order: UTF-16 sorts U+10000 (surrogate pair `D800 DC00`) *before* U+E000 and U+FFFF, while UTF-8 byte order sorts it *after* both. This is an easy mistake in languages whose native string comparison operates on UTF-16 code units (Java's `String.compareTo`, JavaScript, C#), and an implementation that sorts or binary-searches object fields that way will silently miss fields in canonical data (see [SPARK-58949] for an instance of this). None of the existing variant examples contain field names outside the BMP, so such implementations currently pass this test suite. ### What's added Two examples encoding the same 12-field object whose field names straddle the surrogate range (`$ 0 A a ~ U+00A2 U+20AC U+E000 U+FFFF U+10000 U+1F600 U+10FFFF`): * `object_unicode_keys_sorted` — sorted metadata dictionary (`sorted_strings = 1`), field ids in dictionary order; exercises the sorted-dictionary binary-search fast path. * `object_unicode_keys_unsorted` — deliberately scrambled dictionary (`sorted_strings = 0`), so object field ids are non-monotonic and readers must compare the referenced field name bytes rather than assume id order. Each field's value is a short string spelling its field name's code point (e.g. `"U+FFFF"`), so a lookup that lands on the wrong field is self-evident. The files are generated directly from the spec by the new standalone `regen_unicode_keys.py` script (not `regen.py`, which relies on Spark), and were verified to round-trip with an independent decoder. `data_dictionary.json` and the README are updated accordingly; this also fixes a pre-existing trailing comma that made `data_dictionary.json` invalid JSON. [Variant spec]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md [SPARK-58949]: https://issues.apache.org/jira/browse/SPARK-58949 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
