M-Tesla opened a new pull request, #24939: URL: https://github.com/apache/datafusion/pull/24939
## Which issue does this PR close? - Closes #24933 Related: [apache/sedona-db#1231](https://github.com/apache/sedona-db/issues/1231), [apache/sedona-db#1226](https://github.com/apache/sedona-db/pull/1226). This is the uncorrelated physical-plan counterpart of the earlier outer-reference metadata fix in #17524 / #17422. ## Rationale for this change UDFs that distinguish Arrow extension types from their storage types (for example spatial predicates such as `ST_Intersects`) need `ARROW:extension:name` on every argument. Uncorrelated scalar subqueries kept that metadata in the logical plan, but physical planning built a `ScalarSubqueryExpr` from only the data type and nullability. The synthesized physical field had empty metadata, so queries like `WHERE udf(col, (SELECT geometry FROM t WHERE id = 1))` failed even though the equivalent join form worked. ## What changes are included in this PR? - `ScalarSubqueryExpr` now stores the output `FieldRef` (name `scalar_subquery`, original type/nullability, and metadata) via `new_with_metadata`. The existing `new` constructor is unchanged and still produces a field with empty metadata. - Physical lowering copies metadata from the logical subquery output field while still using `Expr::nullable` so zero-row subqueries remain nullable. - Protobuf encoding adds an additive `metadata` map on `PhysicalScalarSubqueryExprNode` so plan round-trips keep extension metadata. ## What is the testing strategy for this PR? - Unit test `scalar_subquery_preserves_output_field_metadata` in `planner.rs` reproduces the drop during physical lowering. - Unit test `return_field_preserves_extension_metadata` and an updated proto round-trip in `scalar_subquery.rs`. - End-to-end regression `test_extension_metadata_preserve_in_uncorrelated_scalar_subquery` in `user_defined_scalar_functions.rs`, based on the issue reproducer. The existing EXISTS-subquery metadata test still passes. ## Are there any user-facing changes? Additive only: `ScalarSubqueryExpr::new_with_metadata` and an optional protobuf `metadata` map (older payloads decode as empty metadata). Existing `new(data_type, nullable, ...)` keeps working. Queries whose UDFs inspect argument field metadata now see the subquery's original extension metadata in the physical plan. Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
