Luka Zdravic created SPARK-60459:
------------------------------------
Summary: Push down filters through ASOF JOIN
Key: SPARK-60459
URL: https://issues.apache.org/jira/browse/SPARK-60459
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
An ASOF JOIN moves no predicate below it. So every row is shuffled and sorted
before a filter runs, and the scans get no data filters for file skipping.
{code:sql}
SELECT * FROM trades t LEFT ASOF JOIN quotes q
MATCH_CONDITION (t.ts >= q.ts) ON t.sym = q.sym AND q.price > 0
WHERE t.qty > 100
{code}
Both filters run at or after the join. These predicates can safely run before
it:
* A WHERE predicate on the left side only.
* An ON predicate on the right side only.
* An ON predicate on the left side only, for an inner join only.
A right-side WHERE predicate must stay after the join. It applies to the
closest match, so moving it below the join would change which right row is the
closest.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
