This is an automated email from the ASF dual-hosted git repository.

HeartSaVioR pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 2ba2e48b8132 [SPARK-60688][SS][SQL] Add the DSv2 connector API for 
external lookup join
2ba2e48b8132 is described below

commit 2ba2e48b8132501d1d06f2618723fa3d9baa5eb7
Author: Jungtaek Lim <[email protected]>
AuthorDate: Wed Nov 4 05:37:36 2026 +0900

    [SPARK-60688][SS][SQL] Add the DSv2 connector API for external lookup join
    
    ### What changes were proposed in this pull request?
    
    An external lookup join enriches a stream with a static table in a 
key-value or OLTP store: each micro-batch looks up only the join keys it needs 
instead of scanning the table 
([SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0),
 [dev](https://lists.apache.org/thread/7s6tgy9cm18jnzykr8htrtrw4nxyrhp1)). This 
PR implements [SPARK-60688](https://issues.apache.org/jira/browse/SPARK-60688), 
part of [SPARK-60031](https://issues.apache.org/jir [...]
    
    All `Evolving`, `since 4.4.0`, in `connector.read` unless noted:
    
    | Piece | Role |
    |---|---|
    | `TableCapability.EXTERNAL_LOOKUP` (`connector.catalog`) | The table has a 
parameterized lookup access path |
    | `SupportsExternalLookup` | `ScanBuilder` mix-in: `Predicate[] 
selectLookupPredicates(Predicate[])` |
    | `LookupParameter` | The typed `?` in an offered predicate |
    | `Scan.toExternalLookup()` | Like `toBatch()`; the default throws 
`DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED` |
    | `ExternalLookup` | `maxParameterRowsPerRequest()`, 
`createReaderFactory()` |
    | `ExternalLookupReaderFactory` | `Serializable`; creates readers on 
executors |
    | `ExternalLookupReader` | `lookupBatch(InternalRow[])` returns a cursor; 
`currentMetricsValues()` |
    | `ExternalLookupResultCursor` | `next()`, `inputOrdinal()`, `get()` |
    
    **Why the contracts look this way**
    - **Identity matching.** Same-shaped candidates must stay distinct. 
Identity, rather than `equals()` or stable ids, does that with the standard 
`Predicate` type:
      ```
      ON t.id = s.a AND t.id = s.b + 1  ->  offered [id = ? (s.a), id = ? (s.b 
+ 1)]
      returning the second binds s.b + 1; Spark evaluates t.id = s.a itself
      ```
    - **Positional parameter rows.** There is one `LookupParameter` per offered 
predicate, so field i of a parameter row is the i-th returned predicate's 
value; no per-row maps.
    - **Complete, parameter-only results.** Spark does not re-evaluate selected 
predicates, and it may reuse a result for equal parameter rows.
    - **Distinct rows under Spark equality.** Spark sends each equal value 
once, carrying the first value seen:
      ```
      -0.0, NaN(bits ..01), 0.0, NaN(bits ..02)  ->  [(-0.0), (NaN ..01)]
      'LOOKUP', 'lookup' under UTF8_LCASE         ->  [('LOOKUP')]
      ```
    - **Offered candidates.** Spark offers `=`, `<=>`, `<`, `<=`, `>`, `>=` 
comparisons, and the `NamedReference` is always `children()[0]`. When the query 
writes the lookup column on the right, the operator is flipped, so `s.ts < 
t.event_time` is offered as `event_time > ?`.
    - **The selection survives pruning.** With a streaming `s`, `SELECT s.*, 
t.name FROM s JOIN t ON t.id = s.id` selects `[id = ?]`, then prunes `id`, so 
the read schema includes `name`; lookups are still by `id`. Spark passes only 
whole top-level columns to `pruneColumns`, never nested fields.
    - **Read schema.** `readSchema()` lists top-level table columns, each at 
most once, in any order. It includes every column passed to `pruneColumns`, or 
every table column if the builder was not pruned.
    - **Reader lifecycle.** Readers are created lazily, at most one per input 
partition; a task may hold several (two lookup joins, or after `coalesce`). 
Calls are sequential, with one exception: if the task ends while another thread 
is still consuming the join, Spark may call `close()` on the reader and its 
open cursor during one call that is already in progress, and starts no new call 
afterward; connectors must tolerate that. Spark closes every cursor it obtains, 
including one it has re [...]
    - **Request size.** At most `maxParameterRowsPerRequest()` rows; Spark may 
send fewer.
    - **Failures.** Exceptions fail the task and never count as misses; retries 
and speculation can repeat a request.
    - **Empty selection.** No fallback to a scan join exists, so connectors 
return `[]` rather than throw. The planning rules 
([SPARK-60689](https://issues.apache.org/jira/browse/SPARK-60689)) then fail 
the query with `EXTERNAL_LOOKUP_JOIN_UNSUPPORTED.NO_LOOKUP_PREDICATE`; this PR 
adds only `DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED`.
    - **Joins and capability 
([SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0)
 Q4, Q6, Q8).** Only stream-static joins: the left side is a streaming query 
and the lookup table is on the right side. A table without `BATCH_READ` is 
joined by lookup automatically and cannot be scanned; a table with both needs 
the `EXTERNAL_LOOKUP` hint.
    
    **Differences from 
[SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0)
 Appendix A**
    - The sketch's prose said `selectLookupPredicates` returns "indexes", 
contradicting its own `Predicate[]` signature; this API returns the offered 
instances, matched by identity.
    - `LookupParameter` is `Serializable`, so a reader factory can keep the 
selected predicates.
    - Added contracts the sketch omits: nulls, distinctness, input shape, 
failure, threading and the read schema.
    - `ToStringSQLBuilder` renders `LookupParameter` as `?`, and a null-safe 
comparison with one as `l <=> r`; without it, `describe()` on a candidate 
throws `UNEXPECTED_V2_EXPRESSION`, or prints one parameter three times. A 
null-safe comparison with a literal keeps its expanded form.
    - The names `LookupParameter` and `lookupBatch` are the sketch's.
    
    Out of scope, tracked under 
[SPARK-60031](https://issues.apache.org/jira/browse/SPARK-60031):
    - Planning rules.
    - Execution.
    - The join hint.
    - Lookup-side operators and more predicate shapes.
    - Cache.
    - Predicate pushdown.
    - Python Data Source support.
    
    ### Why are the changes needed?
    
    A stream-static join loads the static side for every micro-batch, even when 
only a few keys match 
([SPIP](https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0)
 Q3). No existing read API sends per-key requests with join-side values.
    
    ### Does this PR introduce _any_ user-facing change?
    
    New public DSv2 interfaces only; no behavior change.
    
    ### How was this patch tested?
    
    On this commit, JDK 17:
    ```
    build/sbt 'catalyst/testOnly *ExternalLookupApiSuite' 'core/testOnly 
*SparkThrowableSuite' \
      'sql/testOnly *V2PredicateSuite' catalyst/scalastyle 
catalyst/Test/scalastyle
    # JUnit XML: ExternalLookupApiSuite 5/5, SparkThrowableSuite 38/38, 
V2PredicateSuite 29/29 passed
    # scalastyle: 0 errors (main and test); sql/Test compiled with the test run
    javac -Xdoclint:html,syntax,reference,accessibility   # 0 issues (8 API 
files)
    build/mvn -pl sql/catalyst checkstyle:check           # checkstyle 11.0.1, 
dev/checkstyle.xml: 0 violations
    ```
    MiMa was not run.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #59908 from HeartSaVioR/SPARK-60688-external-lookup-connector-api.
    
    Authored-by: Jungtaek Lim <[email protected]>
    Signed-off-by: Jungtaek Lim <[email protected]>
    (cherry picked from commit 957a72564e9f552d8cdd17fcdda6e2dad7cb0e23)
    Signed-off-by: Jungtaek Lim <[email protected]>
---
 .../src/main/resources/error/error-conditions.json |  6 ++
 .../sql/connector/catalog/TableCapability.java     | 12 ++-
 .../spark/sql/connector/read/ExternalLookup.java   | 35 ++++++++
 .../sql/connector/read/ExternalLookupReader.java   | 71 ++++++++++++++++
 .../read/ExternalLookupReaderFactory.java          | 36 ++++++++
 .../connector/read/ExternalLookupResultCursor.java | 47 +++++++++++
 .../spark/sql/connector/read/LookupParameter.java  | 63 ++++++++++++++
 .../org/apache/spark/sql/connector/read/Scan.java  | 28 +++++--
 .../sql/connector/read/SupportsExternalLookup.java | 69 +++++++++++++++
 .../internal/connector/ToStringSQLBuilder.scala    | 13 +++
 .../connector/read/ExternalLookupApiSuite.scala    | 97 ++++++++++++++++++++++
 11 files changed, 471 insertions(+), 6 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json 
b/common/utils/src/main/resources/error/error-conditions.json
index 56a246f7a05c..8b407702d320 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -2387,6 +2387,12 @@
     ],
     "sqlState" : "KD010"
   },
+  "DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED" : {
+    "message" : [
+      "<description>: External lookup is not supported."
+    ],
+    "sqlState" : "0A000"
+  },
   "DATA_SOURCE_INVALID_RUNTIME_FILTER_ATTRIBUTE" : {
     "message" : [
       "The runtime filter attribute <attribute> reported by `<method>` in data 
source scan <scanClass> is invalid for the scan relation output 
<relationOutput>."
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
index e98cf7c11a05..176296f9b80f 100644
--- 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java
@@ -160,5 +160,15 @@ public enum TableCapability {
    *
    * @since 4.3.0
    */
-  SCAN_MERGING
+  SCAN_MERGING,
+
+  /**
+   * Signals that the table has at least one parameterized lookup access path. 
The table must
+   * implement {@link SupportsRead}, its scan builder must implement
+   * {@link org.apache.spark.sql.connector.read.SupportsExternalLookup}, and 
the built scan must
+   * implement {@link 
org.apache.spark.sql.connector.read.Scan#toExternalLookup()}.
+   *
+   * @since 4.4.0
+   */
+  EXTERNAL_LOOKUP
 }
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookup.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookup.java
new file mode 100644
index 000000000000..579c9857bef4
--- /dev/null
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookup.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * The physical representation of an external lookup, returned by {@link 
Scan#toExternalLookup()}.
+ * Spark uses it only on the driver.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookup {
+  /** Returns the maximum number of parameter rows per request, which must be 
positive. */
+  int maxParameterRowsPerRequest();
+
+  /** Returns the factory that Spark serializes and sends to executors. */
+  ExternalLookupReaderFactory createReaderFactory();
+}
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReader.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReader.java
new file mode 100644
index 000000000000..8c0c05b2139b
--- /dev/null
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReader.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Closeable;
+import java.io.IOException;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.connector.metric.CustomTaskMetric;
+
+/**
+ * A task-local client that executes lookup requests.
+ * <ul>
+ *   <li>Calls to a reader and its cursors are sequential, but may come from 
different threads.
+ *   The one exception is the end of the task: if another thread is still 
consuming the join,
+ *   Spark may call {@link #close()} on the reader, and {@code close()} on its 
open cursor, while
+ *   one call from that thread is in progress. Spark starts no new call after 
the close, so
+ *   implementations must tolerate {@code close()} during an in-progress call, 
for example by
+ *   cancelling or failing that call.</li>
+ *   <li>Requests are not snapshot boundaries: results may mix source 
versions.</li>
+ *   <li>Retried or speculative tasks may repeat a request; duplicates must be 
tolerated.</li>
+ *   <li>Implementations should respond to thread interruption if their client 
supports it.</li>
+ * </ul>
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookupReader extends Closeable {
+
+  /**
+   * Looks up {@code uniqueParameters} and returns a cursor over all matches. 
For example:
+   * <pre>{@code
+   *   uniqueParameters: [(1), (2), (3)]
+   *   cursor yields:    (0, a), (0, b), (2, c)    // (inputOrdinal, value row)
+   *   => row 0 has two matches, row 1 is a miss, row 2 has one match
+   * }</pre>
+   * Spark guarantees that {@code uniqueParameters} is non-null and holds 
between 1 and
+   * {@link ExternalLookup#maxParameterRowsPerRequest()} rows, none of them 
null. Field {@code i}
+   * of each row is the value for the {@code i}-th selected predicate, in 
Spark's internal
+   * representation.
+   * Rows are distinct under Spark SQL equality (NaN equals NaN, {@code -0.0} 
equals {@code 0.0},
+   * strings compare by collation). Only {@code <=>} parameters may be null at 
the top level.
+   * <p>
+   * The cursor must be non-null. Each row's results must be complete and 
depend only on the row;
+   * Spark may reuse them for equal rows. The input is read-only and must not 
be kept after the
+   * cursor closes. Spark closes every cursor it obtains, including one it has 
read to the end,
+   * before the next call and before closing this reader.
+   */
+  ExternalLookupResultCursor lookupBatch(InternalRow[] uniqueParameters) 
throws IOException;
+
+  /** Returns cumulative values of the metrics declared by {@link 
Scan#supportedCustomMetrics()}. */
+  default CustomTaskMetric[] currentMetricsValues() {
+    return new CustomTaskMetric[0];
+  }
+}
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReaderFactory.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReaderFactory.java
new file mode 100644
index 000000000000..13a8aa1c94a3
--- /dev/null
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupReaderFactory.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Serializable;
+
+import org.apache.spark.annotation.Evolving;
+
+/**
+ * A serializable factory that Spark sends to executors to create {@link 
ExternalLookupReader}s.
+ * Spark creates at most one reader per input partition that a lookup 
computes, and none if there
+ * is nothing to look up, so one task may hold several readers. If the 
factory, a reader or a
+ * cursor throws, the task fails; a failure is never treated as a miss.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookupReaderFactory extends Serializable {
+  /** Creates a reader on the executor; Spark closes it, best-effort, by the 
end of the task. */
+  ExternalLookupReader createReader();
+}
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupResultCursor.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupResultCursor.java
new file mode 100644
index 000000000000..c6fee41df6ce
--- /dev/null
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/ExternalLookupResultCursor.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Closeable;
+import java.io.IOException;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.catalyst.InternalRow;
+
+/**
+ * A cursor over the results of one {@link ExternalLookupReader#lookupBatch} 
call. Each row is one
+ * match for the parameter row at {@link #inputOrdinal()}: an ordinal may 
repeat, and an absent
+ * ordinal is a miss. If the read schema has no columns, each match is still 
one (empty) row.
+ * <p>
+ * {@link #inputOrdinal()} and {@link #get()} are valid after {@link #next()} 
returns
+ * {@code true}, until the next {@code next} or {@link #close()}. The row may 
be reused; Spark
+ * copies the rows it keeps. Spark closes every cursor, which may be before 
its end.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface ExternalLookupResultCursor extends Closeable {
+  /** Advances to the next result, or returns {@code false} at the end. */
+  boolean next() throws IOException;
+
+  /** Returns the current result's parameter-row index, in {@code [0, 
uniqueParameters.length)}. */
+  int inputOrdinal();
+
+  /** Returns the current value row, whose schema is {@link 
Scan#readSchema()}. */
+  InternalRow get();
+}
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/LookupParameter.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/LookupParameter.java
new file mode 100644
index 000000000000..cc839335e8b8
--- /dev/null
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/LookupParameter.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import java.io.Serializable;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.Expression;
+import org.apache.spark.sql.types.DataType;
+
+/**
+ * A runtime-bound value in an offered lookup predicate, such as the {@code ?} 
in
+ * {@code tenant = ?}; its values arrive in {@link 
ExternalLookupReader#lookupBatch}. Each
+ * candidate has exactly one instance of its own, and equality is identity, so 
same-shaped
+ * candidates are distinct. Instances are serializable, so a connector may 
keep selected
+ * predicates in its {@link ExternalLookupReaderFactory}.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public final class LookupParameter implements Expression, Serializable {
+  private static final long serialVersionUID = 1L;
+  private final DataType dataType;
+
+  public LookupParameter(DataType dataType) {
+    this.dataType = dataType;
+  }
+
+  /** Returns the value's Spark SQL data type; parameter rows use its internal 
representation. */
+  public DataType dataType() {
+    return dataType;
+  }
+
+  @Override
+  public Expression[] children() {
+    return EMPTY_EXPRESSION;
+  }
+
+  @Override
+  public String describe() {
+    return "?";
+  }
+
+  @Override
+  public String toString() {
+    return describe();
+  }
+}
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
index a14f5e12e1f0..94e6bec83c85 100644
--- a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
+++ b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
@@ -33,11 +33,11 @@ import 
org.apache.spark.sql.connector.catalog.TableCapability;
  * A logical representation of a data source scan. This interface is used to 
provide logical
  * information, like what the actual read schema is.
  * <p>
- * This logical representation is shared between batch scan, micro-batch 
streaming scan and
- * continuous streaming scan. Data sources must implement the corresponding 
methods in this
- * interface, to match what the table promises to support. For example, {@link 
#toBatch()} must be
- * implemented, if the {@link Table} that creates this {@link Scan} returns
- * {@link TableCapability#BATCH_READ} support in its {@link 
Table#capabilities()}.
+ * This logical representation is shared between batch scan, micro-batch 
streaming scan,
+ * continuous streaming scan and external lookup. Data sources must implement 
the corresponding
+ * methods in this interface, to match what the table promises to support. For 
example,
+ * {@link #toBatch()} must be implemented, if the {@link Table} that creates 
this {@link Scan}
+ * returns {@link TableCapability#BATCH_READ} support in its {@link 
Table#capabilities()}.
  * </p>
  *
  * @since 3.0.0
@@ -81,6 +81,24 @@ public interface Scan {
       "DATA_SOURCE_BATCH_SCAN_NOT_SUPPORTED", Map.of("description", 
description()));
   }
 
+  /**
+   * Returns the physical representation of this scan for external lookup. The 
default throws
+   * {@code DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED}; data sources must 
override it if the table
+   * reports {@link TableCapability#EXTERNAL_LOOKUP}. Spark calls it at most 
once, on the driver.
+   * {@link #readSchema()} describes the returned value rows and must list 
only top-level table
+   * columns, each at most once, with the table's name and data type; its 
nullability may be
+   * weaker than the table's. The columns may be in any order. They must 
include every column
+   * passed to {@link SupportsPushDownRequiredColumns#pruneColumns}, or every 
table column if the
+   * builder was not pruned.
+   *
+   * @throws UnsupportedOperationException
+   * @since 4.4.0
+   */
+  default ExternalLookup toExternalLookup() {
+    throw new SparkUnsupportedOperationException(
+      "DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED", Map.of("description", 
description()));
+  }
+
   /**
    * Returns the physical representation of this scan for streaming query with 
micro-batch mode. By
    * default this method throws exception, data sources must overwrite this 
method to provide an
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsExternalLookup.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsExternalLookup.java
new file mode 100644
index 000000000000..178ea2cc3ccb
--- /dev/null
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsExternalLookup.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read;
+
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.NamedReference;
+import org.apache.spark.sql.connector.expressions.filter.Predicate;
+
+/**
+ * A mix-in for a {@link ScanBuilder} that can serve parameterized lookups. 
The builder may also
+ * implement the column-pruning and filter-pushdown mix-ins.
+ *
+ * @since 4.4.0
+ */
+@Evolving
+public interface SupportsExternalLookup extends ScanBuilder {
+  /**
+   * Selects the offered predicates that this source serves as one lookup 
access path. Spark
+   * offers {@code =}, {@code <=>}, {@code <}, {@code <=}, {@code >}, {@code 
>=} comparisons of a
+   * lookup-side {@link NamedReference} with a {@link LookupParameter}; the 
array is never empty. In
+   * every offered comparison the reference is {@code children()[0]} and the 
parameter is
+   * {@code children()[1]}: when the query writes the lookup column on the 
right, Spark flips the
+   * operator, so {@code s.ts < t.event_time} is offered as {@code event_time 
> ?}. For example:
+   * <pre>{@code
+   *   candidates: [tenant = ?, event_time >= ?, event_time < ?, region = ?]
+   *   returned:   [event_time >= ?, tenant = ?]
+   *   => each parameter row is (event_time lower bound, tenant);
+   *      Spark itself evaluates event_time < ? and region = ?
+   * }</pre>
+   * <ul>
+   *   <li>The result must be non-null and hold offered instances, unchanged, 
without nulls or
+   *   duplicates. Spark matches them by identity, not {@code equals}.</li>
+   *   <li>Each offered predicate has exactly one {@link LookupParameter}, so 
field {@code i} of
+   *   every parameter row is the value for the {@code i}-th returned 
predicate. A field may
+   *   appear in several predicates, as range bounds do.</li>
+   *   <li>Each selected predicate must hold with Spark semantics, and results 
must be complete
+   *   for their conjunction. Unsupported names or shapes must not be 
selected.</li>
+   *   <li>An empty result means no access path fits; the query then 
fails.</li>
+   *   <li>The input array and its elements are read-only.</li>
+   * </ul>
+   * Spark calls this method only on builders of lookup scans, at most once, 
after any filter
+   * pushdown and before column pruning. Besides filter pushdown and column 
pruning, Spark calls
+   * no pushdown method on a lookup builder (aggregate, limit, top-N, offset, 
sample, join or
+   * variant). Spark passes only whole top-level columns to {@code 
pruneColumns}, never nested
+   * fields. The selection must survive {@code pruneColumns} and {@link 
#build()}. For example,
+   * with a streaming {@code s}:
+   * <pre>{@code
+   *   SELECT s.*, t.name FROM s JOIN t ON t.id = s.id
+   *   => selected [id = ?], then pruneColumns(<name>), so readSchema() 
includes name;
+   *      the connector must still look up by id
+   * }</pre>
+   */
+  Predicate[] selectLookupPredicates(Predicate[] candidates);
+}
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
index 2d6c2256b7fa..816a35ed1ab0 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/ToStringSQLBuilder.scala
@@ -21,6 +21,7 @@ import org.apache.spark.sql.connector.expressions.Expression
 import org.apache.spark.sql.connector.expressions.GeneralScalarExpression
 import org.apache.spark.sql.connector.expressions.GetArrayItem
 import org.apache.spark.sql.connector.expressions.VariantGet
+import org.apache.spark.sql.connector.read.LookupParameter
 import org.apache.spark.sql.connector.util.V2ExpressionSQLBuilder
 
 /**
@@ -40,6 +41,17 @@ class ToStringSQLBuilder extends V2ExpressionSQLBuilder with 
Serializable {
     s"""$funcName($distinct${inputs.mkString(", ")})"""
   }
 
+  // A parameter has no value to repeat, so the null-safe expansion would 
print one parameter as
+  // three markers and misstate how many parameter fields the predicate 
carries.
+  override protected def visitBinaryComparison(
+      name: String, le: Expression, re: Expression): String = {
+    if (name == "<=>" && (le.isInstanceOf[LookupParameter] || 
re.isInstanceOf[LookupParameter])) {
+      inputToSQL(le) + " <=> " + inputToSQL(re)
+    } else {
+      super.visitBinaryComparison(name, le, re)
+    }
+  }
+
   override protected def visitGetArrayItem(getArrayItem: GetArrayItem): String 
= {
     s"${getArrayItem.childArray.toString}[${getArrayItem.ordinal.toString}]"
   }
@@ -58,6 +70,7 @@ class ToStringSQLBuilder extends V2ExpressionSQLBuilder with 
Serializable {
   // until StackOverflowError. Render it as a function call instead.
   override protected def visitUnexpectedExpr(expr: Expression): String = expr 
match {
     case e: GeneralScalarExpression => visitSQLFunction(e.name(), e.children())
+    case p: LookupParameter => p.describe()
     case _ => super.visitUnexpectedExpr(expr)
   }
 }
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/read/ExternalLookupApiSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/read/ExternalLookupApiSuite.scala
new file mode 100644
index 000000000000..f3546b563a18
--- /dev/null
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/read/ExternalLookupApiSuite.scala
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.read
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{SparkConf, SparkFunSuite, 
SparkUnsupportedOperationException}
+import org.apache.spark.serializer.JavaSerializer
+import org.apache.spark.sql.connector.expressions.{Expression, FieldReference, 
LiteralValue}
+import org.apache.spark.sql.connector.expressions.filter.Predicate
+import org.apache.spark.sql.types.{IntegerType, StructType}
+
+class ExternalLookupApiSuite extends SparkFunSuite {
+
+  private def offer(name: String): Predicate =
+    new Predicate(name, Array[Expression](
+      FieldReference.column("id"), new LookupParameter(IntegerType)))
+
+  private val offeredPredicate: Predicate = offer("=")
+
+  private def javaRoundTrip[T: ClassTag](value: T): T = {
+    val serializer = new JavaSerializer(new SparkConf()).newInstance()
+    serializer.deserialize[T](serializer.serialize(value))
+  }
+
+  test("a lookup parameter is a leaf rendered as ?") {
+    val parameter = new LookupParameter(IntegerType)
+    assert(parameter.children().isEmpty)
+    assert(parameter.references().isEmpty)
+    assert(parameter.describe() === "?")
+    assert(offeredPredicate.describe() === "id = ?")
+    assert(offeredPredicate.toString === "id = ?")
+  }
+
+  test("a null-safe candidate renders its parameter once; a literal comparison 
still expands") {
+    assert(offer("<=>").describe() === "id <=> ?")
+    val reversed = new Predicate("<=>", Array[Expression](
+      new LookupParameter(IntegerType), FieldReference.column("id")))
+    assert(reversed.describe() === "? <=> id")
+    assert(offer(">=").describe() === "id >= ?")
+    val literal = new Predicate("<=>", Array[Expression](
+      FieldReference.column("id"), LiteralValue(1, IntegerType)))
+    assert(literal.describe() ===
+      "((id IS NOT NULL AND 1 IS NOT NULL AND id = 1) OR (id IS NULL AND 1 IS 
NULL))")
+  }
+
+  test("candidates of the same shape are distinct because each has its own 
parameter") {
+    val first = offer("=")
+    val second = offer("=")
+    assert(first.equals(first))
+    assert(second.equals(second))
+    assert(!first.equals(second))
+    assert(!first.children()(1).equals(second.children()(1)))
+  }
+
+  test("an offered predicate and a reader factory that retains it are Java 
serializable") {
+    val predicate = javaRoundTrip(offeredPredicate)
+    assert(predicate.describe() === "id = ?")
+    assert(predicate.children()(1).asInstanceOf[LookupParameter].dataType() 
=== IntegerType)
+
+    val factory = 
javaRoundTrip(RetainingLookupReaderFactory(Seq(offeredPredicate)))
+    assert(factory.selected.map(_.describe()) === Seq("id = ?"))
+    assert(factory.selected.head.children()(1).isInstanceOf[LookupParameter])
+  }
+
+  test("a scan without toExternalLookup reports the named error") {
+    val scan = new Scan {
+      override def readSchema(): StructType = new StructType()
+      override def description(): String = "TestScan"
+    }
+    checkError(
+      exception = intercept[SparkUnsupportedOperationException] { 
scan.toExternalLookup() },
+      condition = "DATA_SOURCE_EXTERNAL_LOOKUP_NOT_SUPPORTED",
+      parameters = Map("description" -> "TestScan"))
+  }
+}
+
+/** A reader factory that keeps the connector's selected predicates for use on 
executors. */
+case class RetainingLookupReaderFactory(selected: Seq[Predicate])
+  extends ExternalLookupReaderFactory {
+  override def createReader(): ExternalLookupReader = throw new 
UnsupportedOperationException()
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to