This is an automated email from the ASF dual-hosted git repository.

peter-toth pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new 61eefc89b9a2 [SPARK-60752][4.0][SQL] Share a V2 scan's planned input 
partitions between the copies of its BatchScanExec
61eefc89b9a2 is described below

commit 61eefc89b9a2db88539fc6530ec6798e1d3a3747
Author: Peter Toth <[email protected]>
AuthorDate: Fri Nov 6 20:21:13 2026 +0100

    [SPARK-60752][4.0][SQL] Share a V2 scan's planned input partitions between 
the copies of its BatchScanExec
    
    ### What changes were proposed in this pull request?
    
    Backport of #60030 to branch-4.0, in the shape of the branch-4.3, 
branch-4.2 and branch-4.1 backports #60125, #60172 and #60173. The fix is the 
same as on master, but its shape keeps the signature of `BatchScanExec` in this 
maintenance branch:
    - `BatchScanExec` keeps its `scan` field. Each node holds a `PlannedScan` 
in a private field instead, built from `scan`.
    - `makeCopy` and `clone` hand the node's `PlannedScan` over to a copy over 
the same `Scan`. The rules that rewrite the node do that through `makeCopy`. 
`clone` needs its own override, because `TreeNode.clone` does not go through 
`makeCopy`.
    - Any other new node plans its own, including one the case-class `copy` 
makes. Nothing in Spark copies an executable `BatchScanExec` with `copy`.
    - `PlannedScan` is package private here.
    
    A `PlannedScan` holds the `Scan`, with the `Batch`, its reader factory and 
the input partitions planned from it once, when a node first asks for them. So 
the copies read the partitions the original node planned, with the reader 
factory of the `Batch` that planned them. A pushed runtime filter still replans 
from a new `Batch`, as before.
    
    Nothing is planned earlier than before.
    - The first node that asks plans the partitions, as `inputPartitions` did.
    - A copy reuses them instead of planning again, so a connector's 
`planInputPartitions()` runs less often.
    
    Handing mutable state over in `makeCopy` and `clone` is what 
`InMemoryRelation` does with its `statsOfPlanToCache`.
    
    The branch-4.1 commit applies here with these changes:
    - `BatchScanExec` adds only `SparkPlan` to its imports. `PlannedScan` comes 
after `StoragePartitionJoinParams`, which this branch defines in the same file.
    - `UnsupportedColumnarModeCatalog` overrides this branch's `createTable`, 
which has no constraints, and builds its `InMemoryTable` from a schema.
    
    Like the branch-4.1 backport, compared with the branch-4.2 commit:
    - `BatchScanExec` keeps this branch's `spjParams` field and 
`keyGroupedPartitioning` override. The fix touches neither.
    - `checkStoragePartitionedJoin` runs each case with 
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` on and off. With it on, 
this branch hides most of the bug, see below.
    - `UnsupportedColumnarModeCatalog` copies no pushed filters into its scan, 
since this branch's in-memory scan has none.
    - `SparkPlanSuite` gets only the new test. A test comment no longer 
mentions `GroupPartitionsExec`, which this branch does not have.
    
    ### Why are the changes needed?
    
    A storage-partitioned join can fail when the build side of a DPP subquery 
has a runtime filter of its own. With AQE off:
    1. A DPP subquery that does not reuse a broadcast plans its build side 
again. Its `BatchScanExec` wraps the same `Scan` instance as the join's scan on 
that side.
    2. The build side has a runtime filter of its own, a DPP filter from 
another join. The DPP subquery runs first, and its `filter()` call prunes the 
shared `Scan`. The runtime filtering contracts allow that.
    3. The join's `BatchScanExec` is a copy made after planning. 
`ReuseExchangeAndSubquery` made it when it replaced a subquery in the node's 
runtime filters with a reused one. The copy plans its input partitions again, 
from the already pruned `Scan`.
    4. So the copy reports fewer partitions than the join was planned for.
    
    On this branch, as on branch-4.1, the scan pads its partitions to the 
common partition values the join pushed down, when 
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` is on (the default). 
That hides the missing partitions of the copy. The join fails with `Can't zip 
RDDs with unequal numbers of partitions` in two cases:
    - With `pushPartValues` off, e.g. `List(7, 3)`.
    - When the build side's filter prunes every split, also with 
`pushPartValues` on. The copy then reports no `KeyGroupedPartitioning`, so 
nothing is padded, e.g. `List(7, 0)`.
    
    For example, take `items` partitioned by `id`, `purchases` partitioned by 
`item_id`, ids 1 to 7, and a dimension table `dim(k, flag)` with no row where 
`flag = 2`, over a connector whose `filter()` prunes on `IN`. With DPP on, 
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly=false`, 
`spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio=10`, 
`spark.sql.autoBroadcastJoinThreshold=-1` and AQE off:
    
    ```sql
    SELECT i.id, p.price FROM items i JOIN
      (SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM purchases p
       JOIN dim d ON p.item_id = d.k WHERE d.flag = 2) p
    ON i.id = p.item_id
    ```
    
    fails with `Can't zip RDDs with unequal numbers of partitions: List(7, 0)`.
    
    The scalar subquery shape of #60030 does not fail on this branch, because 
scalar subquery partition filters reach DSv2 runtime filtering only from 
SPARK-56467, in 4.2.0. Its test is kept.
    
    With AQE on, a scan in the default `PARTITION_DEFINED` columnar support 
mode is not affected. AQE asks the copies `supportsColumnar` when it creates 
the result stage, and in that mode this plans their partitions before the DPP 
subquery runs. A scan that declares `SUPPORTED` or `UNSUPPORTED` plans nothing 
there, so it fails the same way with AQE on.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Such a query now returns its rows instead of failing.
    
    ### How was this patch tested?
    
    The tests of #60030, in `KeyGroupedPartitioningSuite` as in #60125. Each 
checks the rows and that the join plans no shuffle, with `pushPartValues` on 
and off.
    - A scalar subquery filter, with one split per key, over 
`InMemoryTableWithV2FilterCatalog`, whose scans prune on `=`. AQE off. It 
passes without the fix on this branch, see above.
    - A DPP filter from a broadcast dimension, with two splits per key, once 
keeping 3 keys and once pruning every split. AQE off.
    - The DPP filter case over a catalog whose scans declare `UNSUPPORTED`, 
with AQE on and off.
    
    Without the fix the last two fail with `Can't zip RDDs with unequal numbers 
of partitions`, the last one with AQE on too.
    
    A new test in `SparkPlanSuite`, over a `Scan` that returns a new `Batch` 
from each `toBatch`. The copies `makeCopy` and `clone` make share the node's 
`Batch` and partitions, and a `makeCopy` over another `Scan` does not. Without 
the fix it fails at the first assertion.
    
    The nested shape also passes, with the right rows and no shuffle, over 
`InMemoryTableCatalog` and `InMemoryTableWithV2FilterCatalog`, with 
`pushPartValues` and AQE on and off, and with or without a dimension match.
    
    On this branch, also ran `SameResultSuite`, `SQLAppStatusListenerSuite`, 
`EnsureRequirementsSuite`, the `DynamicPartitionPruningV2*` suites, 
`DataSourceV2SQLSuiteV2Filter`, `DataSourceV2Suite`, 
`FileBasedDataSourceSuite`, `DataSourceV2OptionSuite`, 
`PruneFileSourcePartitionsSuite`, `OrcV2SchemaPruningSuite`, the 
`GroupBasedRowLevelOperation*` suites, `AvroRowReaderSuite` and `AvroV2Suite`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code with Claude Opus 5.5
    
    Closes #60174 from peter-toth/SPARK-60752-dpp-build-side-shared-scan-4.0.
    
    Authored-by: Peter Toth <[email protected]>
    Signed-off-by: Peter Toth <[email protected]>
---
 .../execution/datasources/v2/BatchScanExec.scala   |  41 ++++-
 .../connector/KeyGroupedPartitioningSuite.scala    | 182 ++++++++++++++++++++-
 .../spark/sql/execution/SparkPlanSuite.scala       |  28 +++-
 3 files changed, 243 insertions(+), 8 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
index 68cb9645b79f..5a361b00f910 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
@@ -29,6 +29,7 @@ import org.apache.spark.sql.catalyst.util.{truncatedString, 
InternalRowComparabl
 import org.apache.spark.sql.connector.catalog.Table
 import org.apache.spark.sql.connector.catalog.functions.Reducer
 import org.apache.spark.sql.connector.read._
+import org.apache.spark.sql.execution.SparkPlan
 import org.apache.spark.util.ArrayImplicits._
 
 /**
@@ -43,7 +44,13 @@ case class BatchScanExec(
     spjParams: StoragePartitionJoinParams = StoragePartitionJoinParams()
   ) extends DataSourceV2ScanExecBase {
 
-  @transient lazy val batch: Batch = if (scan == null) null else scan.toBatch
+  // What this node plans from `scan`, see [[PlannedScan]]. `makeCopy` and 
`clone` hand it over to a
+  // copy over the same `Scan`. Any other new node plans its own, including 
one the case-class
+  // `copy` makes.
+  @transient private var plannedScan: PlannedScan =
+    if (scan == null) null else new PlannedScan(scan)
+
+  def batch: Batch = if (scan == null) null else plannedScan.batch
 
   // TODO: unify the equal/hashCode implementation for all data source v2 
query plans.
   override def equals(other: Any): Boolean = other match {
@@ -57,8 +64,7 @@ case class BatchScanExec(
 
   override def hashCode(): Int = Objects.hashCode(batch, runtimeFilters)
 
-  @transient override lazy val inputPartitions: Seq[InputPartition] =
-    batch.planInputPartitions().toImmutableArraySeq
+  override def inputPartitions: Seq[InputPartition] = 
plannedScan.inputPartitions
 
   @transient private lazy val filteredPartitions: Seq[Seq[InputPartition]] = {
     val dataSourceFilters = runtimeFilters.flatMap {
@@ -158,7 +164,7 @@ case class BatchScanExec(
     }
   }
 
-  override lazy val readerFactory: PartitionReaderFactory = 
batch.createReaderFactory()
+  override def readerFactory: PartitionReaderFactory = 
plannedScan.readerFactory
 
   override lazy val inputRDD: RDD[InternalRow] = {
     val rdd = if (filteredPartitions.isEmpty && outputPartitioning == 
SinglePartition) {
@@ -277,6 +283,19 @@ case class BatchScanExec(
   override def keyGroupedPartitioning: Option[Seq[Expression]] =
     spjParams.keyGroupedPartitioning
 
+  override def makeCopy(newArgs: Array[AnyRef]): SparkPlan = 
sharePlanning(super.makeCopy(newArgs))
+
+  // `TreeNode.clone` does not go through `makeCopy`.
+  override def clone(): SparkPlan = sharePlanning(super.clone())
+
+  private def sharePlanning(node: SparkPlan): SparkPlan = {
+    node match {
+      case c: BatchScanExec if c.scan eq scan => c.plannedScan = plannedScan
+      case _ =>
+    }
+    node
+  }
+
   override def doCanonicalize(): BatchScanExec = {
     this.copy(
       output = output.map(QueryPlan.normalizeExpressions(_, output)),
@@ -321,3 +340,17 @@ case class StoragePartitionJoinParams(
     replicatePartitions: java.lang.Boolean)
 }
 
+/**
+ * A `Scan`, with the `Batch` and the input partitions a [[BatchScanExec]] 
plans from it once, when
+ * a node first asks for them. The node hands it over to the copies `makeCopy` 
and `clone` make of
+ * it, so they read the partitions the first node planned, with a reader 
factory from the `Batch`
+ * that planned them. The partitions have to be the planned ones, because the 
`Scan` can be shared
+ * with another node whose runtime filter prunes it first, e.g. the build side 
of a DPP subquery
+ * (SPARK-60752). A pushed runtime filter replans them from a new `Batch`, but 
keeps the planned
+ * `Batch`'s reader factory.
+ */
+private[v2] final class PlannedScan(val scan: Scan) {
+  lazy val batch: Batch = scan.toBatch
+  lazy val inputPartitions: Seq[InputPartition] = 
batch.planInputPartitions().toImmutableArraySeq
+  lazy val readerFactory: PartitionReaderFactory = batch.createReaderFactory()
+}
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index 56282dc5965a..34cf2cf86ad4 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -25,12 +25,12 @@ import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.{Literal, TransformExpression}
 import org.apache.spark.sql.catalyst.plans.physical
 import org.apache.spark.sql.catalyst.plans.physical.KeyGroupedPartitioning
-import org.apache.spark.sql.connector.catalog.{Column, Identifier, 
InMemoryTableCatalog}
+import org.apache.spark.sql.connector.catalog.{CatalogV2Implicits, 
CatalogV2Util, Column, Identifier, InMemoryTable, InMemoryTableCatalog, 
InMemoryTableWithV2FilterCatalog, Table}
 import org.apache.spark.sql.connector.catalog.functions._
-import org.apache.spark.sql.connector.distributions.Distributions
+import org.apache.spark.sql.connector.distributions.{Distribution, 
Distributions}
 import org.apache.spark.sql.connector.expressions._
 import org.apache.spark.sql.connector.expressions.Expressions._
-import org.apache.spark.sql.connector.read.HasPartitionKey
+import org.apache.spark.sql.connector.read.{HasPartitionKey, Scan, ScanBuilder}
 import org.apache.spark.sql.execution.{RDDScanExec, SparkPlan}
 import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
 import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
@@ -40,6 +40,7 @@ import org.apache.spark.sql.functions.{col, max}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.internal.SQLConf._
 import org.apache.spark.sql.types._
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
 
 class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase {
   private val functions = Seq(
@@ -514,6 +515,139 @@ class KeyGroupedPartitioningSuite extends 
DistributionAndOrderingSuiteBase {
     }
   }
 
+  /**
+   * Runs `f` under the confs of a DPP subquery that reuses no broadcast, and 
drops the tables of
+   * `catalog` after.
+   */
+  private def withDppSubqueryConfs(catalog: InMemoryTableCatalog)(f: => Unit): 
Unit = {
+    withSQLConf(
+        SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+      try f finally catalog.clearTables()
+    }
+  }
+
+  /** Fills `items` and `purchases` with the ids 1 to 7, with `rowsPerKey` 
rows in each purchase. */
+  private def insertDppSubqueryRows(ns: String, rowsPerKey: Int): Unit = {
+    sql(s"INSERT INTO $ns.$items VALUES " + (1 to 7).map(i =>
+      s"($i, 'x', 10.0, cast('2020-01-01' as timestamp))").mkString(", "))
+    sql(s"INSERT INTO $ns.$purchases VALUES " + (1 to 7).flatMap(i =>
+      Seq.fill(rowsPerKey)(s"($i, $i.0, cast('2020-01-01' as 
timestamp))")).mkString(", "))
+  }
+
+  /**
+   * Checks the rows of `query`, and that its join plans no shuffle, with AQE 
off unless `aqe` says
+   * otherwise. With AQE on, the copies of the join's scan plan their 
partitions when the result
+   * stage is created and asks `supportsColumnar`, before the DPP subquery 
prunes the shared `Scan`.
+   * That holds for a scan in the default `PARTITION_DEFINED` columnar support 
mode.
+   *
+   * Each case runs with partition values pushed down and without. With them 
pushed, this branch's
+   * scan pads its partitions to the common partition values of the join, 
which hides a copy that
+   * planned fewer partitions, unless the build side's filter prunes every 
split.
+   */
+  private def checkStoragePartitionedJoin(
+      query: String,
+      expected: Seq[Row],
+      aqe: Seq[Boolean] = Seq(false)): Unit = {
+    for (enabled <- aqe; pushPartValues <- Seq(true, false)) {
+      withSQLConf(
+          SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enabled.toString,
+          SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key -> 
pushPartValues.toString) {
+        val df = sql(query)
+        checkAnswer(df, expected)
+        val plan = df.queryExecution.executedPlan
+        assert(collectShuffles(plan).isEmpty,
+          s"The join should stay storage-partitioned:\n$plan")
+      }
+    }
+  }
+
+  test("SPARK-60752: a DPP subquery's build side does not change the layout of 
the scan it " +
+    "shares with the join") {
+    // A DPP subquery that does not reuse a broadcast plans its build side 
again, over the same
+    // `Scan` as the join's `purchases` scan. A scalar subquery filter on 
`item_id` is a runtime
+    // filter of both, and the subquery runs it first, which prunes the shared 
`Scan`. The join's
+    // scan is a copy made after planning, when the scalar subquery is 
replaced by a reused one. Had
+    // it planned its partitions again, it would report fewer of them than 
`items`, and the join
+    // would fail with "Can't zip RDDs". The V2 filter catalog's scans prune 
on this predicate.
+    val catalogName = "v2_filter_cat"
+    withSQLConf(
+        s"spark.sql.catalog.$catalogName" -> 
classOf[InMemoryTableWithV2FilterCatalog].getName) {
+      val v2FilterCatalog = 
spark.sessionState.catalogManager.catalog(catalogName)
+        .asInstanceOf[InMemoryTableCatalog]
+      val ns = s"$catalogName.ns"
+      withDppSubqueryConfs(v2FilterCatalog) {
+        sql(s"CREATE TABLE $ns.$items (id BIGINT, name STRING, price FLOAT, " +
+          "arrive_time TIMESTAMP) PARTITIONED BY (id)")
+        sql(s"CREATE TABLE $ns.$purchases (item_id BIGINT, price FLOAT, time 
TIMESTAMP) " +
+          "PARTITIONED BY (item_id)")
+        insertDppSubqueryRows(ns, rowsPerKey = 1)
+        checkStoragePartitionedJoin(
+          s"SELECT i.id, p.price FROM $ns.$items i JOIN $ns.$purchases p " +
+            s"ON i.id = p.item_id WHERE i.id = " +
+            s"(SELECT max(item_id) FROM $ns.$purchases WHERE price > 4.0)",
+          Seq(Row(7L, 7.0f)))
+      }
+    }
+  }
+
+  test("SPARK-60752: a nested DPP filter on a DPP subquery's build side does 
not change the " +
+    "layout of the scan it shares with the join") {
+    // As above, with a DPP filter from a broadcast dimension as the build 
side's own runtime
+    // filter, which the in-memory scans prune on. Two rows per key give 
`purchases` two splits per
+    // key, which its scan groups by key. With `flag = 2` the build side's 
filter prunes every
+    // split.
+    withDppSubqueryConfs(catalog) {
+      createTable(items, itemsColumns, Array(identity("id")))
+      createTable(purchases, purchasesColumns, Array(identity("item_id")))
+      insertDppSubqueryRows("testcat.ns", rowsPerKey = 2)
+      createTable("dim", Array(Column.create("k", LongType), 
Column.create("flag", IntegerType)),
+        Array.empty)
+      sql("INSERT INTO testcat.ns.dim VALUES " +
+        (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+      Seq(1 -> (5 to 7), 2 -> Nil).foreach { case (flag, ids) =>
+        checkStoragePartitionedJoin(
+          s"SELECT i.id, p.price FROM testcat.ns.$items i JOIN " +
+            s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM 
testcat.ns.$purchases p " +
+            s"JOIN testcat.ns.dim d ON p.item_id = d.k WHERE d.flag = $flag) p 
" +
+            "ON i.id = p.item_id",
+          ids.flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))))
+      }
+    }
+  }
+
+  test("SPARK-60752: a DPP subquery's build side does not change the layout of 
a shared scan " +
+    "that declares its columnar support mode, with AQE on") {
+    // A scan that declares its mode plans nothing when `supportsColumnar` 
asks, so with AQE on the
+    // copies of the join's scan plan after the build side's filter too. This 
is the nested shape
+    // above, over such a scan, with a dimension match, since an empty 
broadcast lets AQE drop the
+    // join.
+    val catalogName = "unsupported_columnar_cat"
+    withSQLConf(
+        s"spark.sql.catalog.$catalogName" -> 
classOf[UnsupportedColumnarModeCatalog].getName) {
+      val modeCatalog = spark.sessionState.catalogManager.catalog(catalogName)
+        .asInstanceOf[UnsupportedColumnarModeCatalog]
+      val ns = s"$catalogName.ns"
+      withDppSubqueryConfs(modeCatalog) {
+        createTable(items, itemsColumns, Array(identity("id")), catalog = 
modeCatalog)
+        createTable(purchases, purchasesColumns, Array(identity("item_id")),
+          catalog = modeCatalog)
+        insertDppSubqueryRows(ns, rowsPerKey = 2)
+        createTable("dim", Array(Column.create("k", LongType), 
Column.create("flag", IntegerType)),
+          Array.empty, catalog = modeCatalog)
+        sql(s"INSERT INTO $ns.dim VALUES " +
+          (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+        checkStoragePartitionedJoin(
+          s"SELECT i.id, p.price FROM $ns.$items i JOIN " +
+            s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM 
$ns.$purchases p " +
+            s"JOIN $ns.dim d ON p.item_id = d.k WHERE d.flag = 1) p ON i.id = 
p.item_id",
+          (5 to 7).flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))),
+          aqe = Seq(true, false))
+      }
+    }
+  }
+
   test("partitioned join: join with two partition keys and matching & sorted 
partitions") {
     val items_partitions = Array(bucket(8, "id"), days("arrive_time"))
     createTable(items, itemsColumns, items_partitions)
@@ -4252,3 +4386,45 @@ class KeyGroupedPartitioningSuite extends 
DistributionAndOrderingSuiteBase {
     }
   }
 }
+
+/**
+ * An in-memory catalog whose scans declare `UNSUPPORTED` as their columnar 
support mode, so
+ * `supportsColumnar` answers without planning the scan's partitions.
+ */
+class UnsupportedColumnarModeCatalog extends InMemoryTableCatalog {
+  import CatalogV2Implicits._
+
+  override def createTable(
+      ident: Identifier,
+      columns: Array[Column],
+      partitions: Array[Transform],
+      properties: java.util.Map[String, String],
+      distribution: Distribution,
+      ordering: Array[SortOrder],
+      requiredNumPartitions: Option[Int],
+      advisoryPartitionSize: Option[Long],
+      distributionStrictlyRequired: Boolean,
+      numRowsPerSplit: Int): Table = {
+    val table = new InMemoryTable(s"$name.${ident.quoted}",
+        CatalogV2Util.v2ColumnsToStructType(columns), partitions, properties, 
distribution,
+        ordering, requiredNumPartitions, advisoryPartitionSize, 
distributionStrictlyRequired,
+        numRowsPerSplit) {
+      override def newScanBuilder(options: CaseInsensitiveStringMap): 
ScanBuilder =
+        new InMemoryScanBuilder(schema, options) {
+          // This branch's `InMemoryScanBuilder` has no hook for the scan it 
builds, so the built
+          // scan is copied into one that declares `UNSUPPORTED`.
+          override def build: Scan = {
+            val built = super.build.asInstanceOf[InMemoryBatchScan]
+            new InMemoryBatchScan(
+                built.data, built.readSchema, built.tableSchema, 
built.options) {
+              override def columnarSupportMode(): Scan.ColumnarSupportMode =
+                Scan.ColumnarSupportMode.UNSUPPORTED
+            }
+          }
+        }
+    }
+    tables.put(ident, table)
+    namespaces.putIfAbsent(ident.namespace.toList, Map())
+    table
+  }
+}
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
index aed11badb710..06ee4f0c554c 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
@@ -23,10 +23,11 @@ import org.apache.spark.sql.QueryTest
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference}
 import org.apache.spark.sql.catalyst.plans.logical.Deduplicate
+import org.apache.spark.sql.connector.read.{Batch, InputPartition, 
PartitionReaderFactory, Scan}
 import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.IntegerType
+import org.apache.spark.sql.types.{IntegerType, StructType}
 import org.apache.spark.sql.vectorized.ColumnarBatch
 
 class SparkPlanSuite extends QueryTest with SharedSparkSession {
@@ -146,6 +147,31 @@ class SparkPlanSuite extends QueryTest with 
SharedSparkSession {
       }
     }
   }
+
+  test("SPARK-60752: copies of a BatchScanExec share its Batch and the 
partitions it planned") {
+    // A `Scan` that returns a new `Batch` from each `toBatch`, which a 
connector may do.
+    class NewBatchScan extends Scan {
+      override def readSchema(): StructType = new StructType()
+      override def toBatch: Batch = new Batch {
+        override def planInputPartitions(): Array[InputPartition] = Array(new 
InputPartition {})
+        override def createReaderFactory(): PartitionReaderFactory =
+          throw new UnsupportedOperationException()
+      }
+    }
+    val scan = BatchScanExec(Seq.empty, new NewBatchScan, Seq.empty, table = 
null)
+    def sharesPlanning(copy: SparkPlan): Boolean = copy match {
+      case c: BatchScanExec =>
+        (c.batch eq scan.batch) && (c.inputPartitions eq scan.inputPartitions)
+      case _ => false
+    }
+    val args = scan.productIterator.map(_.asInstanceOf[AnyRef]).toArray
+
+    // Copies through `makeCopy` or `clone` share what the node planned, but 
not over another
+    // `Scan`.
+    assert(sharesPlanning(scan.makeCopy(args)))
+    assert(sharesPlanning(scan.clone()))
+    assert(!sharesPlanning(scan.makeCopy(args.updated(1, new NewBatchScan))))
+  }
 }
 
 case class ColumnarOp(child: SparkPlan) extends UnaryExecNode {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to