geoffreyclaude opened a new pull request, #11771: URL: https://github.com/apache/arrow-rs/pull/11771
# Which issue does this PR close? Closes #11598. # Rationale for this change `RowConverter` promises the same ordering as `lexsort`, but Union rows disagree when sorting descending within a child type or placing logical NULLs across type IDs. This can give different sort and grouping behavior depending on which comparison representation an operator uses. # What changes are included in this PR? - Pass the requested sort options to Union child converters. - Give logical NULLs one canonical byte, independent of their active type ID. Shift non-null type bytes to reserve the NULL sentinels and reverse both type and child ordering for descending sorts. - Decode NULLs through the first field marked nullable in schema order, falling back to the first field when none is marked nullable. Advance exactly one encoded byte for these rows so following columns remain intact. - Document the Union encoding and NULL representation. The related logical-NULL and dense-offset fixes are already present on the base branch through #11643 and #11644; this PR does not duplicate them. # Are these changes tested? Regressions compare row ordering against native comparison and `lexsort` across dense/sparse Unions, all four sort options, sliced inputs, extreme/non-contiguous type IDs, and heterogeneous integer/string children. Additional cases cover Union values inside Struct/List arrays, varying trailing sort keys, non-null value/tag preservation, and canonical roundtrips. A nonnullable-first-field case checks the chosen NULL child. The three new public-API regressions fail on the unchanged base and pass with this fix. # Are there any significant user-facing changes? Encoded Union bytes change. Logical NULLs under different type IDs now compare equal, so decoding cannot retain their original active tags: NULL tags are canonicalized as described above. Non-null tags and values remain preserved. This also means a decoded Union array can compare unequal to its input under Arrow array equality, which distinguishes active type IDs even at NULL positions. The tests make that consequence explicit. No public signatures change; changing encoded bytes is separately permitted by the documented release-to-release format instability. This is a draft to discuss whether this comparison-preserving normalization is the right `RowConverter` contract, and which canonical NULL child policy to adopt. Native comparison already equates cross-tag NULLs, while Union array equality and ordinary copying preserve the tag distinction. Existing dictionary hydration is a precedent for representation normalization, but does not settle the Union policy. Preserving the original NULL tags alongside native comparison equality would require provenance outside the comparison bytes or a different row comparison representation. This PR does not change Union array equality. AI assistance: Codex assisted with the patch adaptation, additional regression tests, review, and this description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
