mbutrovich opened a new pull request, #24090: URL: https://github.com/apache/datafusion/pull/24090
## Which issue does this PR close? - Related to https://github.com/apache/datafusion-comet/issues/4859. No DataFusion issue is filed for this one. ## Rationale for this change When a table declares a nested column narrower than the Parquet file's physical type, DataFusion reads every leaf of the column and drops the extra subfields in memory instead of skipping them at read time. - DefaultPhysicalExprAdapter rewrites the projected column into CAST(col AS narrow_type). - Projection mask derivation only understands literal get_field chains, so the cast's inner column expands to every physical leaf via ProjectionMask::roots. - Comet reported a production query reading 1.35 TB where plain Spark read 30.9 GB for the same pruned ReadSchema (datafusion-comet#4859). - Any embedder that hands DataFusion a pre-pruned schema (Comet, delta-rs, Iceberg integrations) hits the same gap. This is a port of #23398 onto current main. #23398 (stacked on the merged #23396 and #23397, superseding an earlier attempt at #23392) implements the fix and was reviewed favorably, but has merge conflicts against main since a follow-up refactor moved PushdownChecker and PushdownColumns into projection_read_plan.rs, and has four unanswered review comments. This PR reimplements the same approach against current main and resolves those four comments by construction: - Drops the nested_projection_pruning config flag; the clip's total fallback design makes a kill switch unnecessary. - Drops an unreachable zero-name-overlap code path; validate_struct_compatibility already rejects that case during physical planning. - Drops the untested union of a cast clip with a sibling get_field access on the same root, in favor of a full read fallback for that case. - Notes that ListView, LargeListView, and Dictionary wrappers are conservatively left unclipped, a candidate follow up. ## What changes are included in this PR? - New module datafusion/datasource-parquet/src/nested_schema_pruning.rs. clip_for_cast walks the physical and cast target type trees together, matches struct fields by name, recurses through List and LargeList, and returns the kept leaf offsets plus the pruned Arrow type in one pass. The clip is total: maps, dictionaries, wrapper-kind mismatches, and any shape it does not understand keep every leaf, so the worst case is today's full read. - PushdownChecker in projection_read_plan.rs now also collects CastColumnAccess entries for a CastExpr over a plain Column where the cast requires nested struct handling. Collection is opt in and only enabled for projection analysis, so filter pushdown is unchanged. - build_projection_read_plan routes to a new build_read_plan_with_cast_clipping when any cast access survives. It partitions referenced roots into whole column reads, cast clipped reads, get_field only reads, and full read fallbacks, including a fallback for a root reached by both a cast and a separate get_field access. - Fixed a pre-existing bug in the has_struct_columns fast path: it only tested the top level field type against Struct, so a LIST STRUCT root such as events was misclassified as containing no struct and skipped PushdownChecker entirely, meaning cast based clipping never fired for that shape. Replaced with a recursive contains_struct check through List, LargeList, ListView, LargeListView, FixedSizeList, Map, Dictionary, and RunEndEncoded. - No new configuration option. ## Are these changes tested? - 15 unit tests in nested_schema_pruning.rs: struct subset, reordering, leaf promotion, missing field null fill, nested struct in struct, list of struct, two levels of list of struct nesting, maps, dictionaries, wrapper mismatches, and an arrow-rs roundtrip test pinning that ProjectionMask::leaves over a subset of List Struct leaves emits exactly the predicted type. - 2 new unit tests in projection_read_plan.rs, plus the pre-existing struct preservation test, unchanged. - 8 integration tests in datafusion/core/tests/parquet/expr_adapter.rs, asserting identical results and a bytes_scanned drop of more than 2x: list of struct narrowing, top level struct, struct level nullability, get_field on a narrowed struct, mixed whole column and subfield access, filter pushdown enabled, a scan mixing a physically narrow and a wide file, and a regression test built from the ReadSchema and InputSchema shapes reported in datafusion-comet#4859 (a two level list STRUCT column with a dropped struct sibling, a dropped map sibling, and dropped top level columns). - New datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt covering the end to end SQL path. - The existing benchmark (already merged as part of #23397) had an assertion documenting the pre-fix baseline, narrow equals full. That assertion now fails as expected and has been flipped to assert narrow reads less than half of full. Measured for the top level struct case: narrow_schema=1081 bytes vs full_schema=8398289 bytes, matching the physically_narrow floor of 1081 bytes. ## Are there any user-facing changes? No API changes and no new configuration option. Behavior is IO reduction only, results are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
