vranes opened a new pull request, #57159: URL: https://github.com/apache/spark/pull/57159
### What changes were proposed in this pull request? This PR adds `BinBy` to `PushPredicateThroughNonJoin.canPushThrough`, so the generic unary-node pushdown arm relocates deterministic predicates below `BIN BY`. - A predicate on a forwarded pass-through or range column pushes below the operator: `BIN BY` replicates those columns unchanged across every sub-row of an input row, so filtering before binning is equivalent to filtering after, and pushing avoids expanding rows that would be discarded. - A predicate on a scaled DISTRIBUTE or appended column cannot push, and stays above: those are produced attributes with fresh `ExprId`s not in the child output, so `pushDownPredicate` never treats them as a subset of the child. This is safe because of the produced-attributes shape from SPARK-57858. Before that change the output DISTRIBUTE column carried the child's `ExprId`, so a predicate on it would have pushed below and filtered the unscaled value. ### Why are the changes needed? `BIN BY` is row-multiplying, so pushing a predicate on a pass-through / range column below the operator filters input rows before binning and avoids wasted expansion (e.g. `WHERE host = '...'` or a time-range predicate). `Generate` is in the same allowlist for the same reason. ### Does this PR introduce _any_ user-facing change? No. `BIN BY` is gated off by default (`spark.sql.binByRelationOperator.enabled`, SPARK-57440). This is an optimizer-only change and does not alter query results. A predicate pushed below the operator can filter out an inverted-range row before it is processed, so whether `BIN_BY_INVALID_RANGE` is raised may depend on the plan, as with any data-dependent error on a discarded row. ### How was this patch tested? - `FilterPushdownSuite`: a predicate on a pass-through column pushes below `BinBy`; a predicate on a produced (fresh-`ExprId`) appended column stays above. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
