leaves12138 commented on code in PR #10067:
URL: https://github.com/apache/paimon/pull/10067#discussion_r4068086717
##########
paimon-python/pypaimon/data/variant_shredding.py:
##########
@@ -328,7 +332,14 @@ def _build_object_value(fields: List[Tuple[int, bytes]])
-> bytes:
buf.append(0) # offset[0] = 0 (sentinel)
return bytes(buf)
- fields = sorted(fields, key=lambda f: f[0])
+ if key_dict is None:
+ fields = sorted(fields, key=lambda f: f[0])
+ else:
+ id_to_name = {key_id: name for name, key_id in key_dict.items()}
+ fields = sorted(
+ fields,
+ key=lambda f: id_to_name[f[0]].encode('utf-16-be'),
Review Comment:
[P2] Preserve UTF-8 byte ordering when rebuilding Variant objects
The Parquet Variant specification requires object field IDs/offsets to be
ordered by field names using unsigned UTF-8 bytes, not UTF-16 code units.
Current Java `GenericVariantBuilder.FieldEntry.compareTo` and Python
`GenericVariant._finish_writing_object` already use UTF-8 ordering.
A concrete regression is `GenericVariant.from_python({'keep': 0, '\uff21':
1, '\U0001f600': 2})` with only `keep` shredded. The metadata IDs for the
remaining keys are already in the required order, so the base writes `[U+FF21,
U+1F600]`; this change writes `[U+1F600, U+FF21]`. I reproduced the same test
passing against the base and failing against this head. Since
`decompose_variant` and `_decompose_field_bytes` call this helper, this affects
persisted overflow bytes, not just an in-memory presentation order. A
conforming reader using UTF-8 binary search can consequently miss existing keys.
Please sort by `.encode('utf-8')` and add a regression mixing a
supplementary-plane key with a BMP key above U+DFFF, covering both overflow
writing and reconstruction. If this exposes a UTF-16-only assumption in the
Rust reader, that compatibility issue should be fixed there rather than
emitting non-conforming bytes here.
Spec:
https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#object-field-id-order-and-uniqueness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
