peterxcli opened a new pull request, #25013: URL: https://github.com/apache/datafusion/pull/25013
## Which issue does this PR close? Closes #21306. ## Rationale for this change Custom scalar functions that extract struct fields currently miss Parquet leaf pruning and decoder-filter pushdown because these paths recognize `GetFieldFunc` directly. `placement()` describes where a function can execute, but does not identify the field it reads. This draft proposes an explicit field-access capability for review. ## What changes are included in this PR? - Add optional `ScalarUDFImpl::struct_field_access()` describing the source argument and literal field path; forward it through `ScalarUDF` and aliases and implement it for `GetFieldFunc`. - Use the capability in Parquet projection/filter planning and physical schema adaptation, retaining the original function and argument layout. - Preserve conservative fallback for Map/repeated-value traversal, functions without the capability, and explicit casts whose sibling conversions can fail. The capability promises exact struct-field extraction, including ancestor null semantics. It does not cover arbitrary Variant decoding or add row-group statistics pruning. FFI forwarding is deferred; wrapped UDFs retain the default fallback. A separate prerequisite commit fixes two existing needless-borrow lint failures with Rust 1.97. ## What is the testing strategy for this PR? Regression tests use distinct custom UDFs, including reversed arguments and aliases. They cover chained paths, literal dots, null parents/children, reordered and narrowed schemas, missing fields, integer widening, explicit-cast errors, and Map fallback. A baseline red test reads all three leaves; the capability selects only the requested leaf. Decoder metrics verify three rows pruned when enabled and zero when the capability or pushdown is disabled. Before rebasing from `ca5d10b75` onto `cc29ea12a`: - Extended workspace tests: 11,233 Rust tests passed, eight ignored; all 511 SQL logic files passed. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `./dev/rust_lint.sh` passed. Post-rebase Clippy and affected tests are running; results will be added here. Existing `parquet_struct_query` benchmark, same 524,288-row file, ten samples, two execution orders (main / patch, milliseconds): | Query | Main then patch | Patch then main | | --- | ---: | ---: | | Nested equality | 20.074 / 18.283 | 17.084 / 16.725 | | Top-level filter control | 8.672 / 6.809 | 7.484 / 7.114 | | Nested range | 18.004 / 20.202 | 18.792 / 18.003 | These used the unoptimized CI profile on pre-rebase main and patch. They do not establish a repeatable speedup or regression; deterministic leaf masks and decoder metrics establish that the optimization occurred. ## Are there any user-facing changes? An additive Rust UDF API lets custom exact struct-field accessors opt into existing Parquet optimizations. Existing implementations default to no capability. No SQL syntax or configuration changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
