j1wonpark opened a new pull request, #57200: URL: https://github.com/apache/spark/pull/57200
### What changes were proposed in this pull request? `V2ScanRelationPushDown.pushDownLimit` had no case for `Union`, so a `LIMIT` on top of a `UNION ALL` was never pushed to the underlying Data Source V2 scans through `SupportsPushDownLimit`; it fell through to `case other => (other, false)`. Note the plan shape: `LimitPushDown` already pushes a `LocalLimit` into each `Union` branch (operator level), but the source-level limit hint (`pushDownLimit`) stops at the `Union`. This PR adds two cases to `pushDownLimit`: - `case u: Union` — recurse into each branch and push the limit as a **partial** limit, always returning `false` so the outer limit operators are kept. - `case l @ LocalLimit(IntegerLiteral(value), _)` — look through the per-branch `LocalLimit` (e.g. the one inserted by `LimitPushDown`) to reach the scan, pushing `min(value, limit)` since the operator itself may cap the rows below the incoming limit. Results are unchanged: `UNION ALL` does not de-duplicate rows, so the union of branches each returning at most `limit` rows still contains the first `limit` rows of the whole union. This mirrors the existing `Union` case in `LimitPushDown`. ### Why are the changes needed? `SELECT ... FROM a UNION ALL SELECT ... FROM b LIMIT n` never delivers the `LIMIT` to the scans, so each source does more work than necessary. Any Data Source V2 connector implementing `SupportsPushDownLimit` benefits from receiving the limit per branch — e.g. for JDBC, each branch's pushed-down SQL can then carry `LIMIT n`. ### Does this PR introduce _any_ user-facing change? No. Query results are identical (partial push keeps the limit operators). The only observable difference is that the pushed limit now appears on each `UNION ALL` branch's scan (e.g. `PushedLimit: LIMIT n` in the plan, or a `LIMIT` clause in the JDBC SQL), reducing work at the source. ### How was this patch tested? Added a unit test to `JDBCV2Suite`: for `SELECT name FROM h2.test.employee UNION ALL SELECT name FROM h2.test.employee LIMIT 1`, it asserts that both branch scans report `pushedDownOperators.limit == Some(1)`, that the outer `GlobalLimit` remains in the optimized plan (the push is partial), and that the query returns 1 row. Ran the full `JDBCV2Suite` (82 tests) — no regression. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
