Jungtaek Lim created SPARK-60694:
------------------------------------
Summary: Push predicates through external lookup joins
Key: SPARK-60694
URL: https://issues.apache.org/jira/browse/SPARK-60694
Project: Spark
Issue Type: Task
Components: SQL, Structured Streaming
Affects Versions: 4.4.0
Reporter: Jungtaek Lim
Part of SPARK-60031. `ExternalLookupJoin` is not a `Join`, so
`PushPredicateThroughJoin` does not apply to it.
In scope:
- A standalone rule, `PushDownPredicatesThroughExternalLookupJoin`, that
matches only `ExternalLookupJoin`.
- It moves only deterministic conjuncts that cannot throw, following
`PushPredicateThroughJoin`'s semantics per join type:
- stream-side filters move below the join;
- lookup-side conjuncts move onto the lookup side, where they reach the
connector or are evaluated on its results.
- Conjuncts without references stay where they are.
- No existing join rule is changed.
Out of scope:
- Join reordering.
- Constraint inference.
- Outer-join elimination for lookup joins.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
