[
https://issues.apache.org/jira/browse/SPARK-60691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-60691:
---------------------------------
Description:
Part of SPARK-60031.
In scope:
- The `EXTERNAL_LOOKUP` hint in DataFrame and SQL forms. Without parameters it
marks the plan it annotates, as join strategy hints do; with relation names it
marks those relations.
- A dual-capability table (`EXTERNAL_LOOKUP` and `BATCH_READ`) becomes a lookup
join only with the hint; without it, it keeps the ordinary join.
- The lookup side is found through hint markers, aliases and views, so SQL and
catalog table references run.
- A join that must be a lookup join but reaches its table through any other
operator, such as a projection or a filter, fails with `LOOKUP_PLAN` when the
DataFrame is built.
- A user join hint is never silently overridden:
- a physical strategy hint on a join over a lookup-only table fails with
`CONFLICTING_JOIN_STRATEGY`;
- a physical strategy hint together with the `EXTERNAL_LOOKUP` hint on a
dual-capability table fails with
`INVALID_EXTERNAL_LOOKUP_HINT.CONFLICTING_JOIN_STRATEGY`;
- a hinted join must also be stream-static (`STATIC_LEFT_SIDE`).
- Error reporting:
- ineligible hinted joins fail with `JOIN_TYPE`, `MISSING_JOIN_CONDITION` or
`STREAMING_LOOKUP_SIDE`, and a hinted table without the `EXTERNAL_LOOKUP`
capability fails with `TABLE`;
- misplaced or invalid hints fail with `INVALID_EXTERNAL_LOOKUP_HINT`
(`NOT_ON_LOOKUP_SIDE`, `MULTIPLE_HINTS`, `RELATION_NOT_FOUND`,
`DUPLICATE_RELATION`, `UNSUPPORTED_PARAMETER`);
- a hint attached to no join is dropped with a warning.
- USING joins (`df.join(lookup.hint("external_lookup"), "id")`) run.
Out of scope:
- Lookup sides with projections or filters, including ones that optimization
would fold away (they fail with `LOOKUP_PLAN`).
- Left semi/anti joins, `<=>` and range predicates.
- Hint options, such as cache options.
was:
Part of SPARK-60031.
In scope:
- The `EXTERNAL_LOOKUP` hint in DataFrame and SQL forms, bare or naming
relations.
- A dual-capability table (`EXTERNAL_LOOKUP` and `BATCH_READ`) becomes a lookup
join only with the hint; without it, it keeps the ordinary join.
- The lookup side is found through hint markers, aliases, views, projections
and filters, so SQL and catalog table references run.
- A user join hint is never silently overridden:
- a physical strategy hint attaching to a lookup join fails with
`CONFLICTING_JOIN_STRATEGY`;
- a hinted join must also be stream-static (`STATIC_LEFT_SIDE`).
- Error reporting:
- ineligible hinted joins fail with `JOIN_TYPE`, `MISSING_JOIN_CONDITION` or
`STREAMING_LOOKUP_SIDE`;
- misplaced hints fail with `INVALID_EXTERNAL_LOOKUP_HINT` or
`NOT_ON_LOOKUP_SIDE`;
- a hint attached to no join is dropped with a warning.
- USING joins (`df.join(lookup.hint("external_lookup"), "id")`) run.
Out of scope:
- Lookup sides that keep a projection or filter after optimization (they fail
with `LOOKUP_PLAN`).
- Left semi/anti joins, `<=>` and range predicates.
- Hint options, such as cache options.
> Add the EXTERNAL_LOOKUP join hint for external lookup join
> ----------------------------------------------------------
>
> Key: SPARK-60691
> URL: https://issues.apache.org/jira/browse/SPARK-60691
> Project: Spark
> Issue Type: Task
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> Part of SPARK-60031.
> In scope:
> - The `EXTERNAL_LOOKUP` hint in DataFrame and SQL forms. Without parameters
> it marks the plan it annotates, as join strategy hints do; with relation
> names it marks those relations.
> - A dual-capability table (`EXTERNAL_LOOKUP` and `BATCH_READ`) becomes a
> lookup join only with the hint; without it, it keeps the ordinary join.
> - The lookup side is found through hint markers, aliases and views, so SQL
> and catalog table references run.
> - A join that must be a lookup join but reaches its table through any other
> operator, such as a projection or a filter, fails with `LOOKUP_PLAN` when the
> DataFrame is built.
> - A user join hint is never silently overridden:
> - a physical strategy hint on a join over a lookup-only table fails with
> `CONFLICTING_JOIN_STRATEGY`;
> - a physical strategy hint together with the `EXTERNAL_LOOKUP` hint on a
> dual-capability table fails with
> `INVALID_EXTERNAL_LOOKUP_HINT.CONFLICTING_JOIN_STRATEGY`;
> - a hinted join must also be stream-static (`STATIC_LEFT_SIDE`).
> - Error reporting:
> - ineligible hinted joins fail with `JOIN_TYPE`, `MISSING_JOIN_CONDITION`
> or `STREAMING_LOOKUP_SIDE`, and a hinted table without the `EXTERNAL_LOOKUP`
> capability fails with `TABLE`;
> - misplaced or invalid hints fail with `INVALID_EXTERNAL_LOOKUP_HINT`
> (`NOT_ON_LOOKUP_SIDE`, `MULTIPLE_HINTS`, `RELATION_NOT_FOUND`,
> `DUPLICATE_RELATION`, `UNSUPPORTED_PARAMETER`);
> - a hint attached to no join is dropped with a warning.
> - USING joins (`df.join(lookup.hint("external_lookup"), "id")`) run.
> Out of scope:
> - Lookup sides with projections or filters, including ones that optimization
> would fold away (they fail with `LOOKUP_PLAN`).
> - Left semi/anti joins, `<=>` and range predicates.
> - Hint options, such as cache options.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
