ohad-definity commented on PR #59706: URL: https://github.com/apache/spark/pull/59706#issuecomment-6372271930
cc @sunchao @viirya @peter-toth, you worked on nested column pruning through `transform` (SPARK-57022, #56070), which this PR builds on, so I'd value your review. This lets `SchemaPruning` and `V2ScanRelationPushDown` look through chains of `explode` / `posexplode`: it closes the `TODO(SPARK-34956)` in `GeneratorNestedColumnAliasing` (several fields of one exploded element) and SPARK-47230 (chained explodes, unused generator output). It stays inside the two existing scan pruning rules, adds no expression and no per-row work, and is behind an internal flag (default on) with a fallback to the unpruned plan. Reading order that I found easiest: 1. `GeneratorsOverScan`: which plans are matched. 2. `pruneThroughGeneratorsImpl`: how fields read above each generator become requirements on its input, top-down. 3. `rebuildAbove`: the bottom-up re-resolution against the narrower types. 4. The last commit's `lengthOnly` (the `size()` / `isnotnull()` filters inferred for a non-outer explode). This is joint work with @toms-definity, and builds on @IgorBerman's work on SPARK-47230: his earlier PRs (#52288, #52598, #54283) mapped out the problem space and the shapes that matter, and he brought them into this PR as `SchemaPruningThroughGenerateSuite` (58 query shapes from Taboola's workloads, each checked against the unpruned results), tested it on those workloads, and did a thorough review that caught a real bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
