[
https://issues.apache.org/jira/browse/SPARK-60692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-60692:
---------------------------------
Description:
Part of SPARK-60031.
In scope:
- Lookup sides with projections and filters. The lookup side is also found
through projections and filters, so views and SQL subqueries such as `(SELECT
... FROM t) s` run:
- projections and filters that remain after optimization run as lookup-side
stages;
- ones that optimization folds away, such as `WHERE 1 = 0`, run without
calling the connector;
- a projection or filter that contains a Python UDF, an external UDF or a
subquery fails with `LOOKUP_PLAN` when the DataFrame is built.
- CHAR padding, and the CHAR/VARCHAR read-side checks under standard semantics,
of a lookup-only table run as a lookup-side projection.
- Filters directly over the table are pushed to the scan builder.
- Required columns are pruned by a standalone rule,
`PruneExternalLookupJoinColumns`. `ColumnPruning` is not changed.
- Left semi and left anti joins.
- `<=>` (null-safe) and `<`, `<=`, `>`, `>=` are offered as lookup candidates;
unselected candidates stay residual.
Out of scope:
- Cache.
- Pushing stream-side filters and join-condition conjuncts through the join.
was:
Part of SPARK-60031.
In scope:
- Projections and filters that remain on the lookup side after optimization run
as lookup-side stages.
- Filters directly over the table are pushed to the scan builder.
- Required columns are pruned by a standalone rule,
`PruneExternalLookupJoinColumns`. `ColumnPruning` is not changed.
- Left semi and left anti joins.
- `<=>` (null-safe) and `<`, `<=`, `>`, `>=` are offered as lookup candidates;
unselected candidates stay residual.
Out of scope:
- Cache.
- Pushing stream-side filters and join-condition conjuncts through the join.
> Support lookup-side projections and filters, semi/anti joins and range
> predicates in external lookup join
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60692
> URL: https://issues.apache.org/jira/browse/SPARK-60692
> Project: Spark
> Issue Type: Task
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> Part of SPARK-60031.
> In scope:
> - Lookup sides with projections and filters. The lookup side is also found
> through projections and filters, so views and SQL subqueries such as `(SELECT
> ... FROM t) s` run:
> - projections and filters that remain after optimization run as lookup-side
> stages;
> - ones that optimization folds away, such as `WHERE 1 = 0`, run without
> calling the connector;
> - a projection or filter that contains a Python UDF, an external UDF or a
> subquery fails with `LOOKUP_PLAN` when the DataFrame is built.
> - CHAR padding, and the CHAR/VARCHAR read-side checks under standard
> semantics, of a lookup-only table run as a lookup-side projection.
> - Filters directly over the table are pushed to the scan builder.
> - Required columns are pruned by a standalone rule,
> `PruneExternalLookupJoinColumns`. `ColumnPruning` is not changed.
> - Left semi and left anti joins.
> - `<=>` (null-safe) and `<`, `<=`, `>`, `>=` are offered as lookup
> candidates; unselected candidates stay residual.
> Out of scope:
> - Cache.
> - Pushing stream-side filters and join-condition conjuncts through the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
