[
https://issues.apache.org/jira/browse/SPARK-60765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhan updated SPARK-60765:
------------------------------
Description:
DSv2 sources can already push filters, aggregates, and a few join types, but a
query that joins, aggregates, or unions tables from the same database still
pulls the raw rows into Spark. The database has an engine that could run that
work, and the network transfer is often the expensive part.
For example, a full outer join of orders and customers, a GROUP BY with
COUNT(DISTINCT), and a UNION with a third table from that same database should
go to the source as one query. Spark should get back only the result rows.
SPARK-52187 added the first join push-down. This SPIP extends that to the
missing join types, set operations, and window functions. A source can decline
any push, and Spark runs that part locally.
SPIP doc:
https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit
was:
SPIP document:
https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit
Extends DSv2 push-down past the single-table operators, building on SPARK-52187.
A source with its own engine can take joins, set operations, and window
functions
and return only the result rows. Sources can decline any push, and Spark falls
back to local execution.
> SPIP: Extend DSv2 push-down with join, set operation, and window function
> support
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60765
> URL: https://issues.apache.org/jira/browse/SPARK-60765
> Project: Spark
> Issue Type: Epic
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Jing Zhan
> Priority: Major
>
> DSv2 sources can already push filters, aggregates, and a few join types, but
> a query that joins, aggregates, or unions tables from the same database still
> pulls the raw rows into Spark. The database has an engine that could run that
> work, and the network transfer is often the expensive part.
> For example, a full outer join of orders and customers, a GROUP BY with
> COUNT(DISTINCT), and a UNION with a third table from that same database
> should go to the source as one query. Spark should get back only the result
> rows. SPARK-52187 added the first join push-down. This SPIP extends that to
> the missing join types, set operations, and window functions. A source can
> decline any push, and Spark runs that part locally.
> SPIP doc:
> https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
