This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new fbe3f9bc9d9b [SPARK-60752][4.3][SQL] Share a V2 scan's planned input 
partitions between the copies of its BatchScanExec
fbe3f9bc9d9b is described below

commit fbe3f9bc9d9b11cd30bd1460a6052e3857280e68
Author: Peter Toth <[email protected]>
AuthorDate: Thu Nov 5 08:41:59 2026 -0800

    [SPARK-60752][4.3][SQL] Share a V2 scan's planned input partitions between 
the copies of its BatchScanExec
    
    ### What changes were proposed in this pull request?
    
    Backport of #60030 to branch-4.3. The fix is the same, but its shape keeps 
the signature of `BatchScanExec` in this maintenance branch:
    - `BatchScanExec` keeps its `scan` field. Each node holds a `PlannedScan` 
in a private field instead, built from `scan`.
    - `makeCopy` and `clone` hand the node's `PlannedScan` over to a copy over 
the same `Scan`. The rules that rewrite the node do that through `makeCopy`. 
`clone` needs its own override, because `TreeNode.clone` does not go through 
`makeCopy`.
    - Any other new node plans its own, including one the case-class `copy` 
makes. Nothing in Spark copies an executable `BatchScanExec` with `copy`.
    - `PlannedScan` is package private here.
    
    A `PlannedScan` holds the `Scan`, with the `Batch`, its reader factory and 
the input partitions planned from it once, when a node first asks for them. So 
the copies read the partitions the original node planned, with the reader 
factory of the `Batch` that planned them. A pushed runtime filter still replans 
from a new `Batch`, as before.
    
    Nothing is planned earlier than before.
    - The first node that asks plans the partitions, as `inputPartitions` did.
    - A copy reuses them instead of planning again, so a connector's 
`planInputPartitions()` runs less often.
    
    Handing mutable state over in `makeCopy` and `clone` is what 
`InMemoryRelation` does with its `statsOfPlanToCache`.
    
    ### Why are the changes needed?
    
    A storage-partitioned join can fail when the build side of a DPP subquery 
has a runtime filter of its own. With AQE off:
    1. A DPP subquery that does not reuse a broadcast plans its build side 
again. Its `BatchScanExec` wraps the same `Scan` instance as the join's scan on 
that side.
    2. The build side has a runtime filter of its own. It can be a DPP filter 
from another join, or a scalar subquery filter routed there by SPARK-56467. The 
DPP subquery runs first, and its `filter()` call prunes the shared `Scan`. The 
runtime filtering contracts allow that.
    3. The join's `BatchScanExec` is a copy made after planning. 
`ReuseExchangeAndSubquery` made it when it replaced a subquery in the node's 
runtime filters with a reused one. The copy plans its input partitions again, 
from the already pruned `Scan`.
    4. So the copy reports fewer partitions than the join was planned for, and 
the join fails with `Can't zip RDDs with unequal numbers of partitions`.
    
    For example, take a connector whose `filter()` drops the partitions an `=` 
predicate rules out, with `items` partitioned by `id` and `purchases` 
partitioned by `item_id`, ids 1 to 7 with one split per key. With DPP on, 
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly=false`, 
`spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio=10`, 
`spark.sql.autoBroadcastJoinThreshold=-1` and AQE off:
    
    ```sql
    SELECT i.id, i.name, p.price FROM items i JOIN purchases p ON i.id = 
p.item_id
    WHERE i.id = (SELECT max(item_id) FROM purchases WHERE price > 4.0)
    ```
    
    fails with `Can't zip RDDs with unequal numbers of partitions: List(7, 1)`.
    
    The same happens when `purchases` is first joined with a broadcast 
dimension, whose DPP filter is then the build side's own runtime filter. A 
connector that prunes on `IN` only is enough for that.
    
    With AQE on, a scan in the default `PARTITION_DEFINED` columnar support 
mode is not affected. AQE asks the copies `supportsColumnar` when it creates 
the result stage, and in that mode this plans their partitions before the DPP 
subquery runs. A scan that declares `SUPPORTED` or `UNSUPPORTED` plans nothing 
there, so it fails the same way with AQE on.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Such a query now returns its rows instead of failing.
    
    ### How was this patch tested?
    
    The tests of #60030, adapted to this branch. It has neither the Catalyst 
runtime filter fixture nor the runtime filter test trait, so they are in 
`KeyGroupedPartitioningSuite`. Each checks the rows and that the join plans no 
shuffle.
    - A scalar subquery filter, with one split per key, over 
`InMemoryTableWithV2FilterCatalog`, whose scans prune on `=`. AQE off.
    - A DPP filter from a broadcast dimension, with two splits per key, once 
keeping 3 keys and once pruning every split. AQE off.
    - The DPP filter case over a catalog whose scans declare `UNSUPPORTED`, 
with AQE on and off.
    
    Without the fix each fails with `Can't zip RDDs with unequal numbers of 
partitions`, the last one with AQE on too.
    
    A new test in `SparkPlanSuite`, over a `Scan` that returns a new `Batch` 
from each `toBatch`. The copies `makeCopy` and `clone` make share the node's 
`Batch` and partitions, and a `makeCopy` over another `Scan` does not. Without 
the fix it fails at the first assertion.
    
    Also ran `GroupPartitionsExecSuite`, `SameResultSuite`, 
`SQLAppStatusListenerSuite`, `EnsureRequirementsSuite`, the 
`DynamicPartitionPruningV2*` suites, `DataSourceV2SQLSuiteV2Filter`, 
`DataSourceV2Suite`, `FileBasedDataSourceSuite`, `DataSourceV2OptionSuite`, 
`PruneFileSourcePartitionsSuite`, `OrcV2SchemaPruningSuite`, the 
`GroupBasedRowLevelOperation*` suites, `AvroRowReaderSuite` and `AvroV2Suite`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code with Claude Opus 5.5
    
    Closes #60125 from peter-toth/SPARK-60752-dpp-build-side-shared-scan-4.3.
    
    Authored-by: Peter Toth <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../execution/datasources/v2/BatchScanExec.scala   |  43 ++++-
 .../connector/KeyGroupedPartitioningSuite.scala    | 177 ++++++++++++++++++++-
 .../spark/sql/execution/SparkPlanSuite.scala       |  28 +++-
 3 files changed, 240 insertions(+), 8 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
index 2e3394ac2a08..289d297cc240 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
@@ -29,7 +29,7 @@ import org.apache.spark.sql.connector.catalog.Table
 import org.apache.spark.sql.connector.read._
 import org.apache.spark.sql.connector.write.RowLevelOperation.Command.DELETE
 import org.apache.spark.sql.connector.write.RowLevelOperationTable
-import org.apache.spark.sql.execution.EmptyRDDWithPartitions
+import org.apache.spark.sql.execution.{EmptyRDDWithPartitions, SparkPlan}
 import org.apache.spark.sql.execution.metric.{SQLLastAttemptMetrics, 
SQLMetric, SQLMetrics}
 import org.apache.spark.util.ArrayImplicits._
 
@@ -45,7 +45,13 @@ case class BatchScanExec(
     keyGroupedPartitioning: Option[Seq[Expression]] = None
   ) extends DataSourceV2ScanExecBase {
 
-  @transient lazy val batch: Batch = if (scan == null) null else scan.toBatch
+  // What this node plans from `scan`, see [[PlannedScan]]. `makeCopy` and 
`clone` hand it over to a
+  // copy over the same `Scan`. Any other new node plans its own, including 
one the case-class
+  // `copy` makes.
+  @transient private var plannedScan: PlannedScan =
+    if (scan == null) null else new PlannedScan(scan)
+
+  def batch: Batch = if (scan == null) null else plannedScan.batch
 
   override protected lazy val sparkMetrics: Map[String, SQLMetric] = {
     val name = "number of output rows"
@@ -73,8 +79,7 @@ case class BatchScanExec(
 
   override def hashCode(): Int = Objects.hash(batch, runtimeFilters, 
keyGroupedPartitioning)
 
-  @transient override lazy val inputPartitions: Seq[InputPartition] =
-    batch.planInputPartitions().toImmutableArraySeq
+  override def inputPartitions: Seq[InputPartition] = 
plannedScan.inputPartitions
 
   // Visible for testing
   @transient private[sql] lazy val filteredPartitions: 
Seq[Option[InputPartition]] =
@@ -86,7 +91,7 @@ case class BatchScanExec(
       outputPartitioning,
       inputPartitions)
 
-  override lazy val readerFactory: PartitionReaderFactory = 
batch.createReaderFactory()
+  override def readerFactory: PartitionReaderFactory = 
plannedScan.readerFactory
 
   override lazy val inputRDD: RDD[InternalRow] = {
     val rdd = if (filteredPartitions.isEmpty && outputPartitioning == 
SinglePartition) {
@@ -100,6 +105,19 @@ case class BatchScanExec(
     rdd
   }
 
+  override def makeCopy(newArgs: Array[AnyRef]): SparkPlan = 
sharePlanning(super.makeCopy(newArgs))
+
+  // `TreeNode.clone` does not go through `makeCopy`.
+  override def clone(): SparkPlan = sharePlanning(super.clone())
+
+  private def sharePlanning(node: SparkPlan): SparkPlan = {
+    node match {
+      case c: BatchScanExec if c.scan eq scan => c.plannedScan = plannedScan
+      case _ =>
+    }
+    node
+  }
+
   override def doCanonicalize(): BatchScanExec = {
     this.copy(
       output = output.map(QueryPlan.normalizeExpressions(_, output)),
@@ -121,3 +139,18 @@ case class BatchScanExec(
     s"BatchScan ${table.name()}".trim
   }
 }
+
+/**
+ * A `Scan`, with the `Batch` and the input partitions a [[BatchScanExec]] 
plans from it once, when
+ * a node first asks for them. The node hands it over to the copies `makeCopy` 
and `clone` make of
+ * it, so they read the partitions the first node planned, with a reader 
factory from the `Batch`
+ * that planned them. The partitions have to be the planned ones, because the 
`Scan` can be shared
+ * with another node whose runtime filter prunes it first, e.g. the build side 
of a DPP subquery
+ * (SPARK-60752). A pushed runtime filter replans them from a new `Batch`, but 
keeps the planned
+ * `Batch`'s reader factory.
+ */
+private[v2] final class PlannedScan(val scan: Scan) {
+  lazy val batch: Batch = scan.toBatch
+  lazy val inputPartitions: Seq[InputPartition] = 
batch.planInputPartitions().toImmutableArraySeq
+  lazy val readerFactory: PartitionReaderFactory = batch.createReaderFactory()
+}
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index 4ff0d3142c51..2d34dcfffe15 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -27,11 +27,13 @@ import 
org.apache.spark.sql.catalyst.expressions.{Ascending, AttributeReference,
 import org.apache.spark.sql.catalyst.plans.Inner
 import org.apache.spark.sql.catalyst.plans.physical
 import org.apache.spark.sql.catalyst.plans.physical.KeyedPartitioning
-import org.apache.spark.sql.connector.catalog.{Column, Identifier, 
InMemoryTableCatalog}
+import org.apache.spark.sql.connector.catalog.{CatalogV2Implicits, Column, 
Identifier, InMemoryTable, InMemoryTableCatalog, 
InMemoryTableWithV2FilterCatalog, Table}
+import org.apache.spark.sql.connector.catalog.constraints.Constraint
 import org.apache.spark.sql.connector.catalog.functions._
-import org.apache.spark.sql.connector.distributions.Distributions
+import org.apache.spark.sql.connector.distributions.{Distribution, 
Distributions}
 import org.apache.spark.sql.connector.expressions._
 import org.apache.spark.sql.connector.expressions.Expressions._
+import org.apache.spark.sql.connector.read.{InputPartition, Scan, ScanBuilder}
 import org.apache.spark.sql.execution.{
   ExtendedMode,
   FormattedMode,
@@ -50,6 +52,7 @@ import org.apache.spark.sql.functions.{col, max}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.internal.SQLConf._
 import org.apache.spark.sql.types._
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
 
 class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase 
with ExplainSuiteHelper {
   private val functions = Seq(
@@ -1510,6 +1513,133 @@ class KeyGroupedPartitioningSuite extends 
DistributionAndOrderingSuiteBase with
     }
   }
 
+  /**
+   * Runs `f` under the confs of a DPP subquery that reuses no broadcast, and 
drops the tables of
+   * `catalog` after.
+   */
+  private def withDppSubqueryConfs(catalog: InMemoryTableCatalog)(f: => Unit): 
Unit = {
+    withSQLConf(
+        SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+      try f finally catalog.clearTables()
+    }
+  }
+
+  /** Fills `items` and `purchases` with the ids 1 to 7, with `rowsPerKey` 
rows in each purchase. */
+  private def insertDppSubqueryRows(ns: String, rowsPerKey: Int): Unit = {
+    sql(s"INSERT INTO $ns.$items VALUES " + (1 to 7).map(i =>
+      s"($i, 'x', 10.0, cast('2020-01-01' as timestamp))").mkString(", "))
+    sql(s"INSERT INTO $ns.$purchases VALUES " + (1 to 7).flatMap(i =>
+      Seq.fill(rowsPerKey)(s"($i, $i.0, cast('2020-01-01' as 
timestamp))")).mkString(", "))
+  }
+
+  /**
+   * Checks the rows of `query`, and that its join plans no shuffle, with AQE 
off unless `aqe` says
+   * otherwise. With AQE on, the copies of the join's scan plan their 
partitions when the result
+   * stage is created and asks `supportsColumnar`, before the DPP subquery 
prunes the shared `Scan`.
+   * That holds for a scan in the default `PARTITION_DEFINED` columnar support 
mode.
+   */
+  private def checkStoragePartitionedJoin(
+      query: String,
+      expected: Seq[Row],
+      aqe: Seq[Boolean] = Seq(false)): Unit = {
+    aqe.foreach { enabled =>
+      withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enabled.toString) {
+        val df = sql(query)
+        checkAnswer(df, expected)
+        val plan = df.queryExecution.executedPlan
+        assert(collectShuffles(plan).isEmpty,
+          s"The join should stay storage-partitioned:\n$plan")
+      }
+    }
+  }
+
+  test("SPARK-60752: a DPP subquery's build side does not change the layout of 
the scan it " +
+    "shares with the join") {
+    // A DPP subquery that does not reuse a broadcast plans its build side 
again, over the same
+    // `Scan` as the join's `purchases` scan. A scalar subquery filter on 
`item_id` is a runtime
+    // filter of both, and the subquery runs it first, which prunes the shared 
`Scan`. The join's
+    // scan is a copy made after planning, when the scalar subquery is 
replaced by a reused one. Had
+    // it planned its partitions again, it would report fewer of them than 
`items`, and the join
+    // would fail with "Can't zip RDDs". The V2 filter catalog's scans prune 
on this predicate.
+    val catalogName = "v2_filter_cat"
+    withSQLConf(
+        s"spark.sql.catalog.$catalogName" -> 
classOf[InMemoryTableWithV2FilterCatalog].getName) {
+      val v2FilterCatalog = 
spark.sessionState.catalogManager.catalog(catalogName)
+        .asInstanceOf[InMemoryTableCatalog]
+      val ns = s"$catalogName.ns"
+      withDppSubqueryConfs(v2FilterCatalog) {
+        sql(s"CREATE TABLE $ns.$items (id BIGINT, name STRING, price FLOAT, " +
+          "arrive_time TIMESTAMP) PARTITIONED BY (id)")
+        sql(s"CREATE TABLE $ns.$purchases (item_id BIGINT, price FLOAT, time 
TIMESTAMP) " +
+          "PARTITIONED BY (item_id)")
+        insertDppSubqueryRows(ns, rowsPerKey = 1)
+        checkStoragePartitionedJoin(
+          s"SELECT i.id, p.price FROM $ns.$items i JOIN $ns.$purchases p " +
+            s"ON i.id = p.item_id WHERE i.id = " +
+            s"(SELECT max(item_id) FROM $ns.$purchases WHERE price > 4.0)",
+          Seq(Row(7L, 7.0f)))
+      }
+    }
+  }
+
+  test("SPARK-60752: a nested DPP filter on a DPP subquery's build side does 
not change the " +
+    "layout of the scan it shares with the join") {
+    // As above, with a DPP filter from a broadcast dimension as the build 
side's own runtime
+    // filter, which the in-memory scans prune on. Two rows per key give 
`purchases` two splits per
+    // key, so a GroupPartitionsExec groups that side. With `flag = 2` the 
build side's filter
+    // prunes every split.
+    withDppSubqueryConfs(catalog) {
+      createTable(items, itemsColumns, Array(identity("id")))
+      createTable(purchases, purchasesColumns, Array(identity("item_id")))
+      insertDppSubqueryRows("testcat.ns", rowsPerKey = 2)
+      createTable("dim", Array(Column.create("k", LongType), 
Column.create("flag", IntegerType)),
+        Array.empty)
+      sql("INSERT INTO testcat.ns.dim VALUES " +
+        (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+      Seq(1 -> (5 to 7), 2 -> Nil).foreach { case (flag, ids) =>
+        checkStoragePartitionedJoin(
+          s"SELECT i.id, p.price FROM testcat.ns.$items i JOIN " +
+            s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM 
testcat.ns.$purchases p " +
+            s"JOIN testcat.ns.dim d ON p.item_id = d.k WHERE d.flag = $flag) p 
" +
+            "ON i.id = p.item_id",
+          ids.flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))))
+      }
+    }
+  }
+
+  test("SPARK-60752: a DPP subquery's build side does not change the layout of 
a shared scan " +
+    "that declares its columnar support mode, with AQE on") {
+    // A scan that declares its mode plans nothing when `supportsColumnar` 
asks, so with AQE on the
+    // copies of the join's scan plan after the build side's filter too. This 
is the nested shape
+    // above, over such a scan, with a dimension match, since an empty 
broadcast lets AQE drop the
+    // join.
+    val catalogName = "unsupported_columnar_cat"
+    withSQLConf(
+        s"spark.sql.catalog.$catalogName" -> 
classOf[UnsupportedColumnarModeCatalog].getName) {
+      val modeCatalog = spark.sessionState.catalogManager.catalog(catalogName)
+        .asInstanceOf[UnsupportedColumnarModeCatalog]
+      val ns = s"$catalogName.ns"
+      withDppSubqueryConfs(modeCatalog) {
+        createTable(items, itemsColumns, Array(identity("id")), catalog = 
modeCatalog)
+        createTable(purchases, purchasesColumns, Array(identity("item_id")),
+          catalog = modeCatalog)
+        insertDppSubqueryRows(ns, rowsPerKey = 2)
+        createTable("dim", Array(Column.create("k", LongType), 
Column.create("flag", IntegerType)),
+          Array.empty, catalog = modeCatalog)
+        sql(s"INSERT INTO $ns.dim VALUES " +
+          (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+        checkStoragePartitionedJoin(
+          s"SELECT i.id, p.price FROM $ns.$items i JOIN " +
+            s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM 
$ns.$purchases p " +
+            s"JOIN $ns.dim d ON p.item_id = d.k WHERE d.flag = 1) p ON i.id = 
p.item_id",
+          (5 to 7).flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))),
+          aqe = Seq(true, false))
+      }
+    }
+  }
+
   test("partitioned join: join with two partition keys and matching & sorted 
partitions") {
     val items_partitions = Array(bucket(8, "id"), days("arrive_time"))
     createTable(items, itemsColumns, items_partitions)
@@ -8217,3 +8347,46 @@ class KeyGroupedPartitioningSuite extends 
DistributionAndOrderingSuiteBase with
   }
 
 }
+
+/**
+ * An in-memory catalog whose scans declare `UNSUPPORTED` as their columnar 
support mode, so
+ * `supportsColumnar` answers without planning the scan's partitions.
+ */
+class UnsupportedColumnarModeCatalog extends InMemoryTableCatalog {
+  import CatalogV2Implicits._
+
+  // scalastyle:off argcount
+  override def createTable(
+      ident: Identifier,
+      columns: Array[Column],
+      partitions: Array[Transform],
+      properties: java.util.Map[String, String],
+      distribution: Distribution,
+      ordering: Array[SortOrder],
+      requiredNumPartitions: Option[Int],
+      advisoryPartitionSize: Option[Long],
+      constraints: Array[Constraint],
+      distributionStrictlyRequired: Boolean,
+      numRowsPerSplit: Int): Table = {
+    // scalastyle:on argcount
+    val table = new InMemoryTable(s"$name.${ident.quoted}", columns, 
partitions, properties,
+        constraints, distribution, ordering, requiredNumPartitions, 
advisoryPartitionSize,
+        distributionStrictlyRequired, numRowsPerSplit) {
+      override def newScanBuilder(options: CaseInsensitiveStringMap): 
ScanBuilder =
+        new InMemoryScanBuilder(schema(), options) {
+          override protected def createScan(
+              partitions: Seq[InputPartition],
+              readSchema: StructType,
+              tableSchema: StructType,
+              options: CaseInsensitiveStringMap): BatchScanBaseClass =
+            new InMemoryBatchScan(partitions, readSchema, tableSchema, 
options) {
+              override def columnarSupportMode(): Scan.ColumnarSupportMode =
+                Scan.ColumnarSupportMode.UNSUPPORTED
+            }
+        }
+    }
+    tables.put(ident, table)
+    namespaces.putIfAbsent(ident.namespace.toList, Map())
+    table
+  }
+}
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
index c0f5ad61d72e..dfee04b47173 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
@@ -31,10 +31,11 @@ import org.apache.spark.sql.catalyst.expressions.{
 import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
ExprCode}
 import org.apache.spark.sql.catalyst.plans.logical.Deduplicate
 import org.apache.spark.sql.catalyst.trees.LeafLike
+import org.apache.spark.sql.connector.read.{Batch, InputPartition, 
PartitionReaderFactory, Scan}
 import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{DataType, IntegerType, StringType}
+import org.apache.spark.sql.types.{DataType, IntegerType, StringType, 
StructType}
 import org.apache.spark.sql.vectorized.ColumnarBatch
 import org.apache.spark.util.ThreadUtils
 
@@ -256,6 +257,31 @@ class SparkPlanSuite extends SharedSparkSession {
     assert(canonicalizedTypes == Seq(IntegerType, StringType),
       s"Expected [IntegerType, StringType] but got $canonicalizedTypes")
   }
+
+  test("SPARK-60752: copies of a BatchScanExec share its Batch and the 
partitions it planned") {
+    // A `Scan` that returns a new `Batch` from each `toBatch`, which a 
connector may do.
+    class NewBatchScan extends Scan {
+      override def readSchema(): StructType = new StructType()
+      override def toBatch: Batch = new Batch {
+        override def planInputPartitions(): Array[InputPartition] = Array(new 
InputPartition {})
+        override def createReaderFactory(): PartitionReaderFactory =
+          throw new UnsupportedOperationException()
+      }
+    }
+    val scan = BatchScanExec(Seq.empty, new NewBatchScan, Seq.empty, table = 
null)
+    def sharesPlanning(copy: SparkPlan): Boolean = copy match {
+      case c: BatchScanExec =>
+        (c.batch eq scan.batch) && (c.inputPartitions eq scan.inputPartitions)
+      case _ => false
+    }
+    val args = scan.productIterator.map(_.asInstanceOf[AnyRef]).toArray
+
+    // Copies through `makeCopy` or `clone` share what the node planned, but 
not over another
+    // `Scan`.
+    assert(sharesPlanning(scan.makeCopy(args)))
+    assert(sharesPlanning(scan.clone()))
+    assert(!sharesPlanning(scan.makeCopy(args.updated(1, new NewBatchScan))))
+  }
 }
 
 case class ColumnarOp(child: SparkPlan) extends UnaryExecNode {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to