[
https://issues.apache.org/jira/browse/SPARK-60690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-60690:
---------------------------------
Epic Link: SPARK-60031
> Execute basic stream-static external lookup joins
> -------------------------------------------------
>
> Key: SPARK-60690
> URL: https://issues.apache.org/jira/browse/SPARK-60690
> Project: Spark
> Issue Type: Task
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> Part of SPARK-60031. This ticket registers the planning rules and adds the
> physical operator, the first user-visible behavior.
> In scope:
> - **Wiring:** the analyzer and optimizer register the rules, and
> `ExternalLookupJoinStrategy` plans `ExternalLookupJoinExec`.
> - **Execution:**
> - batched, synchronous requests;
> - parameter rows deduplicated under Spark equality;
> - one-to-many results, with an early stop under `LIMIT`;
> - readers created lazily per input partition and closed on success, failure
> or cancellation;
> - connector metrics reported under `source.<name>`;
> - a connector failure fails the task and is never treated as a miss.
> - **Analysis:**
> - lookup-only `DataFrameReader` sources load as `ExternalLookupRelation`
> and pass analysis;
> - a lookup-only table read outside a lookup join fails with
> `TABLE_OPERATION`.
> - **Other:**
> - CHAR read padding for the lookup relation;
> - a provably empty lookup side, e.g. `WHERE false`, returns its result
> without calling the connector.
> User-visible result:
> - Micro-batch stream-static inner and left outer joins work when the right
> side is **directly** a lookup-only DSv2 table loaded with `DataFrameReader`,
> with `=` keys (composite included).
> - Real-time mode rejects the operator.
> - Batch (static-static) joins over lookup-only tables fail with
> `STATIC_LEFT_SIDE`.
> Out of scope (these fail with a named error, not bugs):
> - SQL and catalog table references, and aliases, views, projections or
> filters above the lookup table (`TABLE_OPERATION`).
> - The `EXTERNAL_LOOKUP` hint, and strategy hints on the lookup side.
> - Left semi/anti joins, `<=>` and range predicates.
> - CHAR lookup columns (`LOOKUP_PLAN`).
> - Cache.
> - Pushing stream-side filters below the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
