[
https://issues.apache.org/jira/browse/SPARK-60691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-60691:
---------------------------------
Epic Link: SPARK-60031
> Add the EXTERNAL_LOOKUP join hint for external lookup join
> ----------------------------------------------------------
>
> Key: SPARK-60691
> URL: https://issues.apache.org/jira/browse/SPARK-60691
> Project: Spark
> Issue Type: Task
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> Part of SPARK-60031.
> In scope:
> - The `EXTERNAL_LOOKUP` hint in DataFrame and SQL forms, bare or naming
> relations.
> - A dual-capability table (`EXTERNAL_LOOKUP` and `BATCH_READ`) becomes a
> lookup join only with the hint; without it, it keeps the ordinary join.
> - The lookup side is found through hint markers, aliases, views, projections
> and filters, so SQL and catalog table references run.
> - A user join hint is never silently overridden:
> - a physical strategy hint attaching to a lookup join fails with
> `CONFLICTING_JOIN_STRATEGY`;
> - a hinted join must also be stream-static (`STATIC_LEFT_SIDE`).
> - Error reporting:
> - ineligible hinted joins fail with `JOIN_TYPE`, `MISSING_JOIN_CONDITION`
> or `STREAMING_LOOKUP_SIDE`;
> - misplaced hints fail with `INVALID_EXTERNAL_LOOKUP_HINT` or
> `NOT_ON_LOOKUP_SIDE`;
> - a hint attached to no join is dropped with a warning.
> - USING joins (`df.join(lookup.hint("external_lookup"), "id")`) run.
> Out of scope:
> - Lookup sides that keep a projection or filter after optimization (they fail
> with `LOOKUP_PLAN`).
> - Left semi/anti joins, `<=>` and range predicates.
> - Hint options, such as cache options.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
