ump45nose opened a new pull request, #26823: URL: https://github.com/apache/datafusion/pull/26823
> [!IMPORTANT] > > **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push. > See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks). ## Which issue does this PR close? - Closes #26810. ## Rationale for this change `intersect_metadata_for_union` keeps a metadata key only when every UNION branch has the same string value. Arrow stores extension-type metadata under `ARROW:extension:metadata` as a JSON document, and different writers (or even the same writer with different formatting) can serialize the same logical metadata with different whitespace or key order. Those branches then disagree as strings, so the key is dropped and the union field loses its extension metadata even though the JSON values are equivalent. ## What changes are included in this PR? - Compare `ARROW:extension:metadata` as parsed JSON (via `serde_json::Value`) when intersecting metadata for UNION; other keys still require byte-identical strings. - Keep the first branch's serialization when the JSON values agree. - Unit tests for equivalent JSON kept and genuinely different JSON dropped. Ideal longer-term would be a typed equality check on the decoded extension metadata (`DFExtensionType`), but there is no such API yet; JSON equality matches the repro in #26810. ## What is the testing strategy for this PR? - `cargo test -p datafusion-expr intersect_metadata --lib` (covers the existing intersection cases plus `extension_metadata_json_equivalent_kept` and `extension_metadata_json_different_dropped`). ## Are there any user-facing changes? Bug fix only: UNION of columns whose `ARROW:extension:metadata` differs only in JSON formatting now retains that metadata. No public API change. ## AI assistance AI tooling helped draft the patch and tests; I reviewed the approach against #26810 and the existing string intersection logic, and ran the tests above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
