Jungtaek Lim created SPARK-60688:
------------------------------------
Summary: Add the DSv2 connector API for external lookup join
Key: SPARK-60688
URL: https://issues.apache.org/jira/browse/SPARK-60688
Project: Spark
Issue Type: New Feature
Components: SQL, Structured Streaming
Affects Versions: 4.4.0
Reporter: Jungtaek Lim
Part of SPARK-60031. This ticket adds only the connector-facing DSv2 API for
external lookup join. No engine code calls it yet, so it has no user-visible
behavior.
In scope:
- `TableCapability.EXTERNAL_LOOKUP`.
- `SupportsExternalLookup`, a `ScanBuilder` mix-in:
`selectLookupPredicates(Predicate[])`.
- `LookupParameter`: the typed `?` in an offered predicate.
- `Scan.toExternalLookup()`, `ExternalLookup`, `ExternalLookupReaderFactory`,
`ExternalLookupReader` and `ExternalLookupResultCursor`.
- The error condition `DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED`, and `?`
rendering for `LookupParameter` in `ToStringSQLBuilder`.
Out of scope:
- Planning rules.
- Execution.
- The join hint.
- Lookup-side operators and more predicate shapes.
- Cache.
- Predicate pushdown.
- Python Data Source support.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
