This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 2ba2e48b8132 [SPARK-60688][SS][SQL] Add the DSv2 connector API for
external lookup join
2ba2e48b8132 is described below
commit 2ba2e48b8132501d1d06f2618723fa3d9baa5eb7
Author: Jungtaek Lim <[email protected]>
AuthorDate: Wed Nov 4 05:37:36 2026 +0900
[SPARK-60688][SS][SQL] Add the DSv2 connector API for external lookup join
### What changes were proposed in this pull request?
An external lookup join enriches a stream with a static table in a
key-value or OLTP store: each micro-batch looks up only the join keys it needs
instead of scanning the table
([SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0),
[dev](https://lists.apache.org/thread/7s6tgy9cm18jnzykr8htrtrw4nxyrhp1)). This
PR implements [SPARK-60688](https://issues.apache.org/jira/browse/SPARK-60688),
part of [SPARK-60031](https://issues.apache.org/jir [...]
All `Evolving`, `since 4.4.0`, in `connector.read` unless noted:
| Piece | Role |
|---|---|
| `TableCapability.EXTERNAL_LOOKUP` (`connector.catalog`) | The table has a
parameterized lookup access path |
| `SupportsExternalLookup` | `ScanBuilder` mix-in: `Predicate[]
selectLookupPredicates(Predicate[])` |
| `LookupParameter` | The typed `?` in an offered predicate |
| `Scan.toExternalLookup()` | Like `toBatch()`; the default throws
`DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED` |
| `ExternalLookup` | `maxParameterRowsPerRequest()`,
`createReaderFactory()` |
| `ExternalLookupReaderFactory` | `Serializable`; creates readers on
executors |
| `ExternalLookupReader` | `lookupBatch(InternalRow[])` returns a cursor;
`currentMetricsValues()` |
| `ExternalLookupResultCursor` | `next()`, `inputOrdinal()`, `get()` |
**Why the contracts look this way**
- **Identity matching.** Same-shaped candidates must stay distinct.
Identity, rather than `equals()` or stable ids, does that with the standard
`Predicate` type:
```
ON t.id = s.a AND t.id = s.b + 1 -> offered [id = ? (s.a), id = ? (s.b
+ 1)]
returning the second binds s.b + 1; Spark evaluates t.id = s.a itself
```
- **Positional parameter rows.** There is one `LookupParameter` per offered
predicate, so field i of a parameter row is the i-th returned predicate's
value; no per-row maps.
- **Complete, parameter-only results.** Spark does not re-evaluate selected
predicates, and it may reuse a result for equal parameter rows.
- **Distinct rows under Spark equality.** Spark sends each equal value
once, carrying the first value seen:
```
-0.0, NaN(bits ..01), 0.0, NaN(bits ..02) -> [(-0.0), (NaN ..01)]
'LOOKUP', 'lookup' under UTF8_LCASE -> [('LOOKUP')]
```
- **Offered candidates.** Spark offers `=`, `<=>`, `<`, `<=`, `>`, `>=`
comparisons, and the `NamedReference` is always `children()[0]`. When the query
writes the lookup column on the right, the operator is flipped, so `s.ts <
t.event_time` is offered as `event_time > ?`.
- **The selection survives pruning.** With a streaming `s`, `SELECT s.*,
t.name FROM s JOIN t ON t.id = s.id` selects `[id = ?]`, then prunes `id`, so
the read schema includes `name`; lookups are still by `id`. Spark passes only
whole top-level columns to `pruneColumns`, never nested fields.
- **Read schema.** `readSchema()` lists top-level table columns, each at
most once, in any order. It includes every column passed to `pruneColumns`, or
every table column if the builder was not pruned.
- **Reader lifecycle.** Readers are created lazily, at most one per input
partition; a task may hold several (two lookup joins, or after `coalesce`).
Calls are sequential, with one exception: if the task ends while another thread
is still consuming the join, Spark may call `close()` on the reader and its
open cursor during one call that is already in progress, and starts no new call
afterward; connectors must tolerate that. Spark closes every cursor it obtains,
including one it has re [...]
- **Request size.** At most `maxParameterRowsPerRequest()` rows; Spark may
send fewer.
- **Failures.** Exceptions fail the task and never count as misses; retries
and speculation can repeat a request.
- **Empty selection.** No fallback to a scan join exists, so connectors
return `[]` rather than throw. The planning rules
([SPARK-60689](https://issues.apache.org/jira/browse/SPARK-60689)) then fail
the query with `EXTERNAL_LOOKUP_JOIN_UNSUPPORTED.NO_LOOKUP_PREDICATE`; this PR
adds only `DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED`.
- **Joins and capability
([SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0)
Q4, Q6, Q8).** Only stream-static joins: the left side is a streaming query
and the lookup table is on the right side. A table without `BATCH_READ` is
joined by lookup automatically and cannot be scanned; a table with both needs
the `EXTERNAL_LOOKUP` hint.
**Differences from
[SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0)
Appendix A**
- The sketch's prose said `selectLookupPredicates` returns "indexes",
contradicting its own `Predicate[]` signature; this API returns the offered
instances, matched by identity.
- `LookupParameter` is `Serializable`, so a reader factory can keep the
selected predicates.
- Added contracts the sketch omits: nulls, distinctness, input shape,
failure, threading and the read schema.
- `ToStringSQLBuilder` renders `LookupParameter` as `?`, and a null-safe
comparison with one as `l <=> r`; without it, `describe()` on a candidate
throws `UNEXPECTED_V2_EXPRESSION`, or prints one parameter three times. A
null-safe comparison with a literal keeps its expanded form.
- The names `LookupParameter` and `lookupBatch` are the sketch's.
Out of scope, tracked under
[SPARK-60031](https://issues.apache.org/jira/browse/SPARK-60031):
- Planning rules.
- Execution.
- The join hint.
- Lookup-side operators and more predicate shapes.
- Cache.
- Predicate pushdown.
- Python Data Source support.
### Why are the changes needed?
A stream-static join loads the static side for every micro-batch, even when
only a few keys match
([SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0)
Q3). No existing read API sends per-key requests with join-side values.
### Does this PR introduce _any_ user-facing change?
New public DSv2 interfaces only; no behavior change.
### How was this patch tested?
On this commit, JDK 17:
```
build/sbt 'catalyst/testOnly *ExternalLookupApiSuite' 'core/testOnly
*SparkThrowableSuite' \
'sql/testOnly *V2PredicateSuite' catalyst/scalastyle
catalyst/Test/scalastyle
# JUnit XML: ExternalLookupApiSuite 5/5, SparkThrowableSuite 38/38,
V2PredicateSuite 29/29 passed
# scalastyle: 0 errors (main and test); sql/Test compiled with the test run
javac -Xdoclint:html,syntax,reference,accessibility # 0 issues (8 API
files)
build/mvn -pl sql/catalyst checkstyle:check # checkstyle 11.0.1,
dev/checkstyle.xml: 0 violations
```
MiMa was not run.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #59908 from HeartSaVioR/SPARK-60688-external-lookup-connector-api.
Authored-by: Jungtaek Lim <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
(cherry picked from commit 957a72564e9f552d8cdd17fcdda6e2dad7cb0e23)
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 6 ++
.../sql/connector/catalog/TableCapability.java | 12 ++-
.../spark/sql/connector/read/ExternalLookup.java | 35 ++++++++
.../sql/connector/read/ExternalLookupReader.java | 71 ++++++++++++++++
.../read/ExternalLookupReaderFactory.java | 36 ++++++++
.../connector/read/ExternalLookupResultCursor.java | 47 +++++++++++
.../spark/sql/connector/read/LookupParameter.java | 63 ++++++++++++++
.../org/apache/spark/sql/connector/read/Scan.java | 28 +++++--
.../sql/connector/read/SupportsExternalLookup.java | 69 +++++++++++++++
.../internal/connector/ToStringSQLBuilder.scala | 13 +++
.../connector/read/ExternalLookupApiSuite.scala | 97 ++++++++++++++++++++++
11 files changed, 471 insertions(+), 6 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 56a246f7a05c..8b407702d320 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -2387,6 +2387,12 @@
],
"sqlState" : "KD010"
},
+ "DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED" : {
+ "message" : [
+ "<description>: External lookup is not supported."
+ ],
+ "sqlState" : "0A000"
+ },
"DATA_SOURCE_INVALID_RUNTIME_FILTER_ATTRIBUTE" : {
"message" : [
"The runtime filter attribute <attribute> reported by `<method>` in data
source scan <scanClass> is invalid for the scan relation output
<relationOutput>."
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
index e98cf7c11a05..176296f9b80f 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
@@ -160,5 +160,15 @@ public enum TableCapability {
*
* @since 4.3.0
*/
- SCAN_MERGING
+ SCAN_MERGING,
+
+ /**
+ * Signals that the table has at least one parameterized lookup access path.
The table must
+ * implement {@link SupportsRead}, its scan builder must implement
+ * {@link org.apache.spark.sql.connector.read.SupportsExternalLookup}, and
the built scan must
+ * implement {@link
org.apache.spark.sql.connector.read.Scan#toExternalLookup()}.
+ *
+ * @since 4.4.0
+ */
+ EXTERNAL_LOOKUP
}
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookup.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookup.java
new file mode 100644
index 000000000000..579c9857bef4
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookup.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * The physical representation of an external lookup, returned by {@link
Scan#toExternalLookup()}.
+ * Spark uses it only on the driver.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookup {
+ /** Returns the maximum number of parameter rows per request, which must be
positive. */
+ int maxParameterRowsPerRequest();
+
+ /** Returns the factory that Spark serializes and sends to executors. */
+ ExternalLookupReaderFactory createReaderFactory();
+}
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReader.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReader.java
new file mode 100644
index 000000000000..8c0c05b2139b
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReader.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Closeable;
+import java.io.IOException;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.connector.metric.CustomTaskMetric;
+
+/**
+ * A task-local client that executes lookup requests.
+ * <ul>
+ * <li>Calls to a reader and its cursors are sequential, but may come from
different threads.
+ * The one exception is the end of the task: if another thread is still
consuming the join,
+ * Spark may call {@link #close()} on the reader, and {@code close()} on its
open cursor, while
+ * one call from that thread is in progress. Spark starts no new call after
the close, so
+ * implementations must tolerate {@code close()} during an in-progress call,
for example by
+ * cancelling or failing that call.</li>
+ * <li>Requests are not snapshot boundaries: results may mix source
versions.</li>
+ * <li>Retried or speculative tasks may repeat a request; duplicates must be
tolerated.</li>
+ * <li>Implementations should respond to thread interruption if their client
supports it.</li>
+ * </ul>
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookupReader extends Closeable {
+
+ /**
+ * Looks up {@code uniqueParameters} and returns a cursor over all matches.
For example:
+ * <pre>{@code
+ * uniqueParameters: [(1), (2), (3)]
+ * cursor yields: (0, a), (0, b), (2, c) // (inputOrdinal, value row)
+ * => row 0 has two matches, row 1 is a miss, row 2 has one match
+ * }</pre>
+ * Spark guarantees that {@code uniqueParameters} is non-null and holds
between 1 and
+ * {@link ExternalLookup#maxParameterRowsPerRequest()} rows, none of them
null. Field {@code i}
+ * of each row is the value for the {@code i}-th selected predicate, in
Spark's internal
+ * representation.
+ * Rows are distinct under Spark SQL equality (NaN equals NaN, {@code -0.0}
equals {@code 0.0},
+ * strings compare by collation). Only {@code <=>} parameters may be null at
the top level.
+ * <p>
+ * The cursor must be non-null. Each row's results must be complete and
depend only on the row;
+ * Spark may reuse them for equal rows. The input is read-only and must not
be kept after the
+ * cursor closes. Spark closes every cursor it obtains, including one it has
read to the end,
+ * before the next call and before closing this reader.
+ */
+ ExternalLookupResultCursor lookupBatch(InternalRow[] uniqueParameters)
throws IOException;
+
+ /** Returns cumulative values of the metrics declared by {@link
Scan#supportedCustomMetrics()}. */
+ default CustomTaskMetric[] currentMetricsValues() {
+ return new CustomTaskMetric[0];
+ }
+}
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReaderFactory.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReaderFactory.java
new file mode 100644
index 000000000000..13a8aa1c94a3
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReaderFactory.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Serializable;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * A serializable factory that Spark sends to executors to create {@link
ExternalLookupReader}s.
+ * Spark creates at most one reader per input partition that a lookup
computes, and none if there
+ * is nothing to look up, so one task may hold several readers. If the
factory, a reader or a
+ * cursor throws, the task fails; a failure is never treated as a miss.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookupReaderFactory extends Serializable {
+ /** Creates a reader on the executor; Spark closes it, best-effort, by the
end of the task. */
+ ExternalLookupReader createReader();
+}
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupResultCursor.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupResultCursor.java
new file mode 100644
index 000000000000..c6fee41df6ce
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupResultCursor.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Closeable;
+import java.io.IOException;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.catalyst.InternalRow;
+
+/**
+ * A cursor over the results of one {@link ExternalLookupReader#lookupBatch}
call. Each row is one
+ * match for the parameter row at {@link #inputOrdinal()}: an ordinal may
repeat, and an absent
+ * ordinal is a miss. If the read schema has no columns, each match is still
one (empty) row.
+ * <p>
+ * {@link #inputOrdinal()} and {@link #get()} are valid after {@link #next()}
returns
+ * {@code true}, until the next {@code next} or {@link #close()}. The row may
be reused; Spark
+ * copies the rows it keeps. Spark closes every cursor, which may be before
its end.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookupResultCursor extends Closeable {
+ /** Advances to the next result, or returns {@code false} at the end. */
+ boolean next() throws IOException;
+
+ /** Returns the current result's parameter-row index, in {@code [0,
uniqueParameters.length)}. */
+ int inputOrdinal();
+
+ /** Returns the current value row, whose schema is {@link
Scan#readSchema()}. */
+ InternalRow get();
+}
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/LookupParameter.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/LookupParameter.java
new file mode 100644
index 000000000000..cc839335e8b8
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/LookupParameter.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Serializable;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.Expression;
+import org.apache.spark.sql.types.DataType;
+
+/**
+ * A runtime-bound value in an offered lookup predicate, such as the {@code ?}
in
+ * {@code tenant = ?}; its values arrive in {@link
ExternalLookupReader#lookupBatch}. Each
+ * candidate has exactly one instance of its own, and equality is identity, so
same-shaped
+ * candidates are distinct. Instances are serializable, so a connector may
keep selected
+ * predicates in its {@link ExternalLookupReaderFactory}.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public final class LookupParameter implements Expression, Serializable {
+ private static final long serialVersionUID = 1L;
+ private final DataType dataType;
+
+ public LookupParameter(DataType dataType) {
+ this.dataType = dataType;
+ }
+
+ /** Returns the value's Spark SQL data type; parameter rows use its internal
representation. */
+ public DataType dataType() {
+ return dataType;
+ }
+
+ @Override
+ public Expression[] children() {
+ return EMPTY_EXPRESSION;
+ }
+
+ @Override
+ public String describe() {
+ return "?";
+ }
+
+ @Override
+ public String toString() {
+ return describe();
+ }
+}
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
index a14f5e12e1f0..94e6bec83c85 100644
--- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
+++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
@@ -33,11 +33,11 @@ import
org.apache.spark.sql.connector.catalog.TableCapability;
* A logical representation of a data source scan. This interface is used to
provide logical
* information, like what the actual read schema is.
* <p>
- * This logical representation is shared between batch scan, micro-batch
streaming scan and
- * continuous streaming scan. Data sources must implement the corresponding
methods in this
- * interface, to match what the table promises to support. For example, {@link
#toBatch()} must be
- * implemented, if the {@link Table} that creates this {@link Scan} returns
- * {@link TableCapability#BATCH_READ} support in its {@link
Table#capabilities()}.
+ * This logical representation is shared between batch scan, micro-batch
streaming scan,
+ * continuous streaming scan and external lookup. Data sources must implement
the corresponding
+ * methods in this interface, to match what the table promises to support. For
example,
+ * {@link #toBatch()} must be implemented, if the {@link Table} that creates
this {@link Scan}
+ * returns {@link TableCapability#BATCH_READ} support in its {@link
Table#capabilities()}.
* </p>
*
* @since 3.0.0
@@ -81,6 +81,24 @@ public interface Scan {
"DATA_SOURCE_BATCH_SCAN_NOT_SUPPORTED", Map.of("description",
description()));
}
+ /**
+ * Returns the physical representation of this scan for external lookup. The
default throws
+ * {@code DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED}; data sources must
override it if the table
+ * reports {@link TableCapability#EXTERNAL_LOOKUP}. Spark calls it at most
once, on the driver.
+ * {@link #readSchema()} describes the returned value rows and must list
only top-level table
+ * columns, each at most once, with the table's name and data type; its
nullability may be
+ * weaker than the table's. The columns may be in any order. They must
include every column
+ * passed to {@link SupportsPushDownRequiredColumns#pruneColumns}, or every
table column if the
+ * builder was not pruned.
+ *
+ * @throws UnsupportedOperationException
+ * @since 4.4.0
+ */
+ default ExternalLookup toExternalLookup() {
+ throw new SparkUnsupportedOperationException(
+ "DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED", Map.of("description",
description()));
+ }
+
/**
* Returns the physical representation of this scan for streaming query with
micro-batch mode. By
* default this method throws exception, data sources must overwrite this
method to provide an
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsExternalLookup.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsExternalLookup.java
new file mode 100644
index 000000000000..178ea2cc3ccb
--- /dev/null
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsExternalLookup.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+
+/**
+ * A mix-in for a {@link ScanBuilder} that can serve parameterized lookups.
The builder may also
+ * implement the column-pruning and filter-pushdown mix-ins.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface SupportsExternalLookup extends ScanBuilder {
+ /**
+ * Selects the offered predicates that this source serves as one lookup
access path. Spark
+ * offers {@code =}, {@code <=>}, {@code <}, {@code <=}, {@code >}, {@code
>=} comparisons of a
+ * lookup-side {@link NamedReference} with a {@link LookupParameter}; the
array is never empty. In
+ * every offered comparison the reference is {@code children()[0]} and the
parameter is
+ * {@code children()[1]}: when the query writes the lookup column on the
right, Spark flips the
+ * operator, so {@code s.ts < t.event_time} is offered as {@code event_time
> ?}. For example:
+ * <pre>{@code
+ * candidates: [tenant = ?, event_time >= ?, event_time < ?, region = ?]
+ * returned: [event_time >= ?, tenant = ?]
+ * => each parameter row is (event_time lower bound, tenant);
+ * Spark itself evaluates event_time < ? and region = ?
+ * }</pre>
+ * <ul>
+ * <li>The result must be non-null and hold offered instances, unchanged,
without nulls or
+ * duplicates. Spark matches them by identity, not {@code equals}.</li>
+ * <li>Each offered predicate has exactly one {@link LookupParameter}, so
field {@code i} of
+ * every parameter row is the value for the {@code i}-th returned
predicate. A field may
+ * appear in several predicates, as range bounds do.</li>
+ * <li>Each selected predicate must hold with Spark semantics, and results
must be complete
+ * for their conjunction. Unsupported names or shapes must not be
selected.</li>
+ * <li>An empty result means no access path fits; the query then
fails.</li>
+ * <li>The input array and its elements are read-only.</li>
+ * </ul>
+ * Spark calls this method only on builders of lookup scans, at most once,
after any filter
+ * pushdown and before column pruning. Besides filter pushdown and column
pruning, Spark calls
+ * no pushdown method on a lookup builder (aggregate, limit, top-N, offset,
sample, join or
+ * variant). Spark passes only whole top-level columns to {@code
pruneColumns}, never nested
+ * fields. The selection must survive {@code pruneColumns} and {@link
#build()}. For example,
+ * with a streaming {@code s}:
+ * <pre>{@code
+ * SELECT s.*, t.name FROM s JOIN t ON t.id = s.id
+ * => selected [id = ?], then pruneColumns(<name>), so readSchema()
includes name;
+ * the connector must still look up by id
+ * }</pre>
+ */
+ Predicate[] selectLookupPredicates(Predicate[] candidates);
+}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
index 2d6c2256b7fa..816a35ed1ab0 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
@@ -21,6 +21,7 @@ import org.apache.spark.sql.connector.expressions.Expression
import org.apache.spark.sql.connector.expressions.GeneralScalarExpression
import org.apache.spark.sql.connector.expressions.GetArrayItem
import org.apache.spark.sql.connector.expressions.VariantGet
+import org.apache.spark.sql.connector.read.LookupParameter
import org.apache.spark.sql.connector.util.V2ExpressionSQLBuilder
/**
@@ -40,6 +41,17 @@ class ToStringSQLBuilder extends V2ExpressionSQLBuilder with
Serializable {
s"""$funcName($distinct${inputs.mkString(", ")})"""
}
+ // A parameter has no value to repeat, so the null-safe expansion would
print one parameter as
+ // three markers and misstate how many parameter fields the predicate
carries.
+ override protected def visitBinaryComparison(
+ name: String, le: Expression, re: Expression): String = {
+ if (name == "<=>" && (le.isInstanceOf[LookupParameter] ||
re.isInstanceOf[LookupParameter])) {
+ inputToSQL(le) + " <=> " + inputToSQL(re)
+ } else {
+ super.visitBinaryComparison(name, le, re)
+ }
+ }
+
override protected def visitGetArrayItem(getArrayItem: GetArrayItem): String
= {
s"${getArrayItem.childArray.toString}[${getArrayItem.ordinal.toString}]"
}
@@ -58,6 +70,7 @@ class ToStringSQLBuilder extends V2ExpressionSQLBuilder with
Serializable {
// until StackOverflowError. Render it as a function call instead.
override protected def visitUnexpectedExpr(expr: Expression): String = expr
match {
case e: GeneralScalarExpression => visitSQLFunction(e.name(), e.children())
+ case p: LookupParameter => p.describe()
case _ => super.visitUnexpectedExpr(expr)
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/read/ExternalLookupApiSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/read/ExternalLookupApiSuite.scala
new file mode 100644
index 000000000000..f3546b563a18
--- /dev/null
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/read/ExternalLookupApiSuite.scala
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{SparkConf, SparkFunSuite,
SparkUnsupportedOperationException}
+import org.apache.spark.serializer.JavaSerializer
+import org.apache.spark.sql.connector.expressions.{Expression, FieldReference,
LiteralValue}
+import org.apache.spark.sql.connector.expressions.filter.Predicate
+import org.apache.spark.sql.types.{IntegerType, StructType}
+
+class ExternalLookupApiSuite extends SparkFunSuite {
+
+ private def offer(name: String): Predicate =
+ new Predicate(name, Array[Expression](
+ FieldReference.column("id"), new LookupParameter(IntegerType)))
+
+ private val offeredPredicate: Predicate = offer("=")
+
+ private def javaRoundTrip[T: ClassTag](value: T): T = {
+ val serializer = new JavaSerializer(new SparkConf()).newInstance()
+ serializer.deserialize[T](serializer.serialize(value))
+ }
+
+ test("a lookup parameter is a leaf rendered as ?") {
+ val parameter = new LookupParameter(IntegerType)
+ assert(parameter.children().isEmpty)
+ assert(parameter.references().isEmpty)
+ assert(parameter.describe() === "?")
+ assert(offeredPredicate.describe() === "id = ?")
+ assert(offeredPredicate.toString === "id = ?")
+ }
+
+ test("a null-safe candidate renders its parameter once; a literal comparison
still expands") {
+ assert(offer("<=>").describe() === "id <=> ?")
+ val reversed = new Predicate("<=>", Array[Expression](
+ new LookupParameter(IntegerType), FieldReference.column("id")))
+ assert(reversed.describe() === "? <=> id")
+ assert(offer(">=").describe() === "id >= ?")
+ val literal = new Predicate("<=>", Array[Expression](
+ FieldReference.column("id"), LiteralValue(1, IntegerType)))
+ assert(literal.describe() ===
+ "((id IS NOT NULL AND 1 IS NOT NULL AND id = 1) OR (id IS NULL AND 1 IS
NULL))")
+ }
+
+ test("candidates of the same shape are distinct because each has its own
parameter") {
+ val first = offer("=")
+ val second = offer("=")
+ assert(first.equals(first))
+ assert(second.equals(second))
+ assert(!first.equals(second))
+ assert(!first.children()(1).equals(second.children()(1)))
+ }
+
+ test("an offered predicate and a reader factory that retains it are Java
serializable") {
+ val predicate = javaRoundTrip(offeredPredicate)
+ assert(predicate.describe() === "id = ?")
+ assert(predicate.children()(1).asInstanceOf[LookupParameter].dataType()
=== IntegerType)
+
+ val factory =
javaRoundTrip(RetainingLookupReaderFactory(Seq(offeredPredicate)))
+ assert(factory.selected.map(_.describe()) === Seq("id = ?"))
+ assert(factory.selected.head.children()(1).isInstanceOf[LookupParameter])
+ }
+
+ test("a scan without toExternalLookup reports the named error") {
+ val scan = new Scan {
+ override def readSchema(): StructType = new StructType()
+ override def description(): String = "TestScan"
+ }
+ checkError(
+ exception = intercept[SparkUnsupportedOperationException] {
scan.toExternalLookup() },
+ condition = "DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED",
+ parameters = Map("description" -> "TestScan"))
+ }
+}
+
+/** A reader factory that keeps the connector's selected predicates for use on
executors. */
+case class RetainingLookupReaderFactory(selected: Seq[Predicate])
+ extends ExternalLookupReaderFactory {
+ override def createReader(): ExternalLookupReader = throw new
UnsupportedOperationException()
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]