vranes opened a new pull request, #57157: URL: https://github.com/apache/spark/pull/57157
### What changes were proposed in this pull request? This PR adds Generate-style column pruning to the logical `BinBy` operator, so a `Project` over `BIN BY` trims forwarded pass-through columns that neither the project nor the operator needs. - `BinBy` gains an `unrequiredChildIndex: Seq[Int]` field (starts as `Nil`, filled by the optimizer) and a `requiredChildOutput` lazy val, mirroring `Generate`. `output` becomes `requiredChildOutput` (with each DISTRIBUTE slot swapped to its scaled produced attribute) `++ appendedAttributes`. - A `ColumnPruning` arm for `Project(_, b: BinBy)` sets `unrequiredChildIndex`. It keeps the range and DISTRIBUTE inputs the kernel reads: they stay in the child even when not forwarded to output (range columns the project does not select drop from output but remain in the child; DISTRIBUTE columns are always required, since they are swapped to their scaled produced attrs in `output`). - `ResolveBinBy` passes `unrequiredChildIndex = Nil` at construction. This is enabled by the produced-attributes shape from SPARK-57858: the scaled DISTRIBUTE columns and appended columns are produced attributes with fresh `ExprId`s, so `BinBy.references` lists exactly the child inputs the kernel reads, and `ColumnPruning` can never prune those away. ### Why are the changes needed? `BIN BY` is row-multiplying, so forwarding unused pass-through columns through the operator wastes work proportional to the fan-out. Pruning them at the child, as `Generate` already does, avoids carrying columns the query does not use. ### Does this PR introduce _any_ user-facing change? No. `BIN BY` is gated off by default (`spark.sql.binByRelationOperator.enabled`, SPARK-57440). This is an optimizer-only change that does not alter query results. ### How was this patch tested? - `ColumnPruningSuite`: a `Project` over `BinBy` prunes the unused pass-through column while keeping the range and DISTRIBUTE inputs in the child, and populates `unrequiredChildIndex`. - `ResolveBinBySuite`: existing analyzer coverage, green after the output-shape change (`unrequiredChildIndex = Nil` leaves `requiredChildOutput == child.output`, so `output` is unchanged when nothing is pruned). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
