[
https://issues.apache.org/jira/browse/SPARK-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539527#comment-17539527
]
Victor Delépine commented on SPARK-19609:
-----------------------------------------
Hey [~hyukjin.kwon],
I can confirm that this is still an issue in Spar 3.2.1, as my team just ran
into this doing a stream/static broadcast join. Would it make sense to reopen
this ticket or create a new one, so that it's on the radar? This could have a
pretty large impact on a lot of pipelines.
And thanks [~jwesteen] for the workaround! That helped us a lot
> Broadcast joins should pushdown join constraints as Filter to the larger
> relation
> ---------------------------------------------------------------------------------
>
> Key: SPARK-19609
> URL: https://issues.apache.org/jira/browse/SPARK-19609
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Nick Dimiduk
> Priority: Major
> Labels: bulk-closed
>
> For broadcast inner-joins, where the smaller relation is known to be small
> enough to materialize on a worker, the set of values for all join columns is
> known and fits in memory. Spark should translate these values into a
> {{Filter}} pushed down to the datasource. The common join condition of
> equality, i.e. {{lhs.a == rhs.a}}, can be written as an {{a in ...}} clause.
> An example of pushing such filters is already present in the form of
> {{IsNotNull}} filters via [~sameerag]'s work on SPARK-12957 subtasks.
> This optimization could even work when the smaller relation does not fit
> entirely in memory. This could be done by partitioning the smaller relation
> into N pieces, applying this predicate pushdown for each piece, and unioning
> the results.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
