[
https://issues.apache.org/jira/browse/SPARK-60459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić reassigned SPARK-60459:
------------------------------------
Assignee: Luka Zdravic
> Push down filters through ASOF JOIN
> -----------------------------------
>
> Key: SPARK-60459
> URL: https://issues.apache.org/jira/browse/SPARK-60459
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Assignee: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
>
> An ASOF JOIN moves no predicate below it. So every row is shuffled and
> sorted before a filter runs, and the scans get no data filters for file
> skipping.
> {code:sql}
> SELECT * FROM trades t LEFT ASOF JOIN quotes q
> MATCH_CONDITION (t.ts >= q.ts) ON t.sym = q.sym AND q.price > 0
> WHERE t.qty > 100
> {code}
> Both filters run at or after the join. These predicates can safely run
> before it:
> * A WHERE predicate on the left side only.
> * An ON predicate on the right side only.
> * An ON predicate on the left side only, for an inner join only.
> A right-side WHERE predicate must stay after the join. It applies to the
> closest match, so moving it below the join would change which right row is
> the closest.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
