Jungtaek Lim created SPARK-60692:
------------------------------------
Summary: Support lookup-side projections and filters, semi/anti
joins and range predicates in external lookup join
Key: SPARK-60692
URL: https://issues.apache.org/jira/browse/SPARK-60692
Project: Spark
Issue Type: Task
Components: SQL, Structured Streaming
Affects Versions: 4.4.0
Reporter: Jungtaek Lim
Part of SPARK-60031.
In scope:
- Projections and filters that remain on the lookup side after optimization run
as lookup-side stages.
- Filters directly over the table are pushed to the scan builder.
- Required columns are pruned by a standalone rule,
`PruneExternalLookupJoinColumns`. `ColumnPruning` is not changed.
- Left semi and left anti joins.
- `<=>` (null-safe) and `<`, `<=`, `>`, `>=` are offered as lookup candidates;
unselected candidates stay residual.
Out of scope:
- Cache.
- Pushing stream-side filters and join-condition conjuncts through the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
