Victor Delépine created SPARK-39753:
---------------------------------------
Summary: Broadcast joins should pushdown join constraints as
Filter to the larger relation
Key: SPARK-39753
URL: https://issues.apache.org/jira/browse/SPARK-39753
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.3.0, 3.2.1, 3.2.0
Reporter: Victor Delépine
SPARK-19609 was bulk-closed a while ago, but not fixed. I've decided to re-open
it here for more visibility, since I believe this bug has a major impact and
that fixing it could drastically improve the performance of many pipelines.
Allow me to paste the initial description again here:
_For broadcast inner-joins, where the smaller relation is known to be small
enough to materialize on a worker, the set of values for all join columns is
known and fits in memory. Spark should translate these values into a {{Filter}}
pushed down to the datasource. The common join condition of equality, i.e.
{{{}lhs.a == rhs.a{}}}, can be written as an {{a in ...}} clause. An example of
pushing such filters is already present in the form of {{IsNotNull}} filters
via_ [~sameerag]{_}'s work on SPARK-12957 subtasks.{_}
_This optimization could even work when the smaller relation does not fit
entirely in memory. This could be done by partitioning the smaller relation
into N pieces, applying this predicate pushdown for each piece, and unioning
the results._
Essentially, when doing a Broadcast join, the smaller side can be used to
filter down the bigger side before performing the join. As of today, the join
will reads all partitions of the bigger side, without pruning partitions
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
