I like this proposal, I think we're at a good point to modernize some of the DSv2 pushdowns.
On Fri, Oct 30, 2026 at 3:03 PM Jing Zhan <[email protected]> wrote: > Hi all, > > I'd like to start a discussion on a SPIP to extend DSv2 operator push-down. > > JIRA: https://issues.apache.org/jira/browse/SPARK-60765 > Design: > https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit > Builds on SPARK-52187 (DSv2 join push-down). > Shepherd: Holden Karau > > Today DSv2 can push filters, column pruning, aggregates, limit/TopN, > sample, and INNER/LEFT/RIGHT joins. Those joins are off unless > spark.sql.optimizer.datasourceV2JoinPushdown is set. A query that uses > other join types, a set operation, or a window function over the same > database still pulls the raw rows into Spark. > > The proposal adds opt-in interfaces, each of which a source can decline: > 1. FULL OUTER / SEMI / ANTI join types > 2. Alias tolerance in join push-down > 3. Make the join push-down config public, and evaluate enabling it by > default > 4. Join + aggregate composition > 5. Set operations (UNION / INTERSECT / EXCEPT) > 6. Window functions > > JDBC/H2 is the reference implementation. This does not cover cross-source > push-down or file-based sources. > > Please reply to this thread. A [VOTE] will follow once the discussion > settles. > > Thanks, > Jing Zhan > -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
