CurtHagenlocher opened a new pull request, #125: URL: https://github.com/apache/parquet-testing/pull/125
Closes #124. ## What Adds two object examples in `variant/`: | case | shape | header byte | widths | |---|---|---|---| | `object_wide_offsets` | 2 fields, >256 bytes of field data | `0x06` | 1-byte field IDs, 2-byte field offsets | | `object_wide_field_ids` | 262 distinct field names; small object under `nested` | `0x12` (nested object) | 2-byte field IDs, 1-byte field offsets | These are the first examples in which an object's `field_id_size` and `field_offset_size` differ. Every existing object example uses 1-byte field IDs and 1-byte field offsets, where the two `value_header` fields hold the same value, so a reader that reads them from each other's bit positions still decodes correctly. arrow-dotnet shipped exactly that bug and validated cleanly against this corpus the whole time (apache/arrow-dotnet#420). `object_wide_field_ids` has 261 fields, which also makes it the first example with `is_large` set to 1 (a 4-byte `num_elements`). ## Verification - `regen.py` was run end to end and reproduces the four new files byte-identically, along with 49 of the 52 files it generates. See below for the three that differ. - Both new examples decode to the expected JSON under an independent spec-literal decoder, which was first cross-checked against the existing `object_nested` and `array_nested` examples. - Both are rejected by a reader that transposes the two `value_header` fields: `object_wide_offsets` fails its end-offset check, `object_wide_field_ids` runs off the end of the buffer. ## Notes for reviewers - The JSON for these two cases is built in Python and substituted into the SQL, since 262 field names and a 300-byte string are not readable as inline literals. - I did not regenerate the rest of the corpus, and `data_dictionary.json` is edited in place rather than rewritten, since the committed file is hand-maintained (it carries a trailing comma, and `primitive_timestampntz` already precedes `primitive_timestamp_nanos`). - `object_wide_field_ids` also covers the first case listed in #78. The 2\*\*16 and 2\*\*24 cases there remain open; their TODOs in `regen.py` now sit under a reference to #78 rather than #77, and are reworded, since the number of distinct fields drives the field ID width rather than the field offset width. ### Two pre-existing regen.py deviations, unrelated to this change Noting these since I ran the script; happy to file them separately if useful. - `primitive_null.metadata` and `primitive_null.value` come out **empty**, which README "Modification 1" already documents as needing a manual fix-up after every regen. - `primitive_timestamp.value` is **timezone dependent**. `'2025-04-16T12:34:56.78'::Timestamp` is interpreted in the Spark session timezone, so the committed file decodes to `2025-04-16 16:34:56Z` (generated under UTC-4) while a run on a US Pacific machine produces `19:34:56Z`. Anyone regenerating the corpus will silently rewrite this file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
