This is an automated email from the ASF dual-hosted git repository.
peter-toth pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new ce007f3417b2 [SPARK-60752][4.1][SQL] Share a V2 scan's planned input
partitions between the copies of its BatchScanExec
ce007f3417b2 is described below
commit ce007f3417b24cccb621ae01eedba6ef1b3969b4
Author: Peter Toth <[email protected]>
AuthorDate: Fri Nov 6 17:20:52 2026 +0100
[SPARK-60752][4.1][SQL] Share a V2 scan's planned input partitions between
the copies of its BatchScanExec
### What changes were proposed in this pull request?
Backport of #60030 to branch-4.1, in the shape of the branch-4.3 and
branch-4.2 backports #60125 and #60172. The fix is the same as on master, but
its shape keeps the signature of `BatchScanExec` in this maintenance branch:
- `BatchScanExec` keeps its `scan` field. Each node holds a `PlannedScan`
in a private field instead, built from `scan`.
- `makeCopy` and `clone` hand the node's `PlannedScan` over to a copy over
the same `Scan`. The rules that rewrite the node do that through `makeCopy`.
`clone` needs its own override, because `TreeNode.clone` does not go through
`makeCopy`.
- Any other new node plans its own, including one the case-class `copy`
makes. Nothing in Spark copies an executable `BatchScanExec` with `copy`.
- `PlannedScan` is package private here.
A `PlannedScan` holds the `Scan`, with the `Batch`, its reader factory and
the input partitions planned from it once, when a node first asks for them. So
the copies read the partitions the original node planned, with the reader
factory of the `Batch` that planned them. A pushed runtime filter still replans
from a new `Batch`, as before.
Nothing is planned earlier than before.
- The first node that asks plans the partitions, as `inputPartitions` did.
- A copy reuses them instead of planning again, so a connector's
`planInputPartitions()` runs less often.
Handing mutable state over in `makeCopy` and `clone` is what
`InMemoryRelation` does with its `statsOfPlanToCache`.
The branch-4.2 commit applies here with these changes:
- `BatchScanExec` keeps this branch's `spjParams` field and
`keyGroupedPartitioning` override. The fix touches neither.
- `checkStoragePartitionedJoin` runs each case with
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` on and off. With it on,
this branch hides most of the bug, see below.
- `UnsupportedColumnarModeCatalog` copies no pushed filters into its scan,
since this branch's in-memory scan has none.
- `SparkPlanSuite` gets only the new test. A test comment no longer
mentions `GroupPartitionsExec`, which this branch does not have.
### Why are the changes needed?
A storage-partitioned join can fail when the build side of a DPP subquery
has a runtime filter of its own. With AQE off:
1. A DPP subquery that does not reuse a broadcast plans its build side
again. Its `BatchScanExec` wraps the same `Scan` instance as the join's scan on
that side.
2. The build side has a runtime filter of its own, a DPP filter from
another join. The DPP subquery runs first, and its `filter()` call prunes the
shared `Scan`. The runtime filtering contracts allow that.
3. The join's `BatchScanExec` is a copy made after planning.
`ReuseExchangeAndSubquery` made it when it replaced a subquery in the node's
runtime filters with a reused one. The copy plans its input partitions again,
from the already pruned `Scan`.
4. So the copy reports fewer partitions than the join was planned for.
On this branch the scan pads its partitions to the common partition values
the join pushed down, when
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` is on (the default).
That hides the missing partitions of the copy. The join fails with `Can't zip
RDDs with unequal numbers of partitions` in two cases:
- With `pushPartValues` off, e.g. `List(7, 3)`.
- When the build side's filter prunes every split, also with
`pushPartValues` on. The copy then reports no `KeyGroupedPartitioning`, so
nothing is padded, e.g. `List(7, 0)`.
For example, take `items` partitioned by `id`, `purchases` partitioned by
`item_id`, ids 1 to 7, and a dimension table `dim(k, flag)` with no row where
`flag = 2`, over a connector whose `filter()` prunes on `IN`. With DPP on,
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly=false`,
`spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio=10`,
`spark.sql.autoBroadcastJoinThreshold=-1` and AQE off:
```sql
SELECT i.id, p.price FROM items i JOIN
(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM purchases p
JOIN dim d ON p.item_id = d.k WHERE d.flag = 2) p
ON i.id = p.item_id
```
fails with `Can't zip RDDs with unequal numbers of partitions: List(7, 0)`.
The scalar subquery shape of #60030 does not fail on this branch, because
scalar subquery partition filters reach DSv2 runtime filtering only from
SPARK-56467, in 4.2.0. Its test is kept.
With AQE on, a scan in the default `PARTITION_DEFINED` columnar support
mode is not affected. AQE asks the copies `supportsColumnar` when it creates
the result stage, and in that mode this plans their partitions before the DPP
subquery runs. A scan that declares `SUPPORTED` or `UNSUPPORTED` plans nothing
there, so it fails the same way with AQE on.
### Does this PR introduce _any_ user-facing change?
Yes. Such a query now returns its rows instead of failing.
### How was this patch tested?
The tests of #60030, in `KeyGroupedPartitioningSuite` as in #60125. Each
checks the rows and that the join plans no shuffle, with `pushPartValues` on
and off.
- A scalar subquery filter, with one split per key, over
`InMemoryTableWithV2FilterCatalog`, whose scans prune on `=`. AQE off. It
passes without the fix on this branch, see above.
- A DPP filter from a broadcast dimension, with two splits per key, once
keeping 3 keys and once pruning every split. AQE off.
- The DPP filter case over a catalog whose scans declare `UNSUPPORTED`,
with AQE on and off.
Without the fix the last two fail with `Can't zip RDDs with unequal numbers
of partitions`, the last one with AQE on too.
A new test in `SparkPlanSuite`, over a `Scan` that returns a new `Batch`
from each `toBatch`. The copies `makeCopy` and `clone` make share the node's
`Batch` and partitions, and a `makeCopy` over another `Scan` does not. Without
the fix it fails at the first assertion.
The nested shape also passes, with the right rows and no shuffle, over
`InMemoryTableCatalog` and `InMemoryTableWithV2FilterCatalog`, with
`pushPartValues` and AQE on and off, and with or without a dimension match.
On this branch, also ran `SameResultSuite`, `SQLAppStatusListenerSuite`,
`EnsureRequirementsSuite`, the `DynamicPartitionPruningV2*` suites,
`DataSourceV2SQLSuiteV2Filter`, `DataSourceV2Suite`,
`FileBasedDataSourceSuite`, `DataSourceV2OptionSuite`,
`PruneFileSourcePartitionsSuite`, `OrcV2SchemaPruningSuite`, the
`GroupBasedRowLevelOperation*` suites, `AvroRowReaderSuite` and `AvroV2Suite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 5.5
Closes #60173 from peter-toth/SPARK-60752-dpp-build-side-shared-scan-4.1.
Authored-by: Peter Toth <[email protected]>
Signed-off-by: Peter Toth <[email protected]>
---
.../execution/datasources/v2/BatchScanExec.scala | 43 ++++-
.../connector/KeyGroupedPartitioningSuite.scala | 185 ++++++++++++++++++++-
.../spark/sql/execution/SparkPlanSuite.scala | 28 +++-
3 files changed, 247 insertions(+), 9 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
index a7f8b3640204..32e82a8032b3 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
@@ -28,7 +28,7 @@ import
org.apache.spark.sql.catalyst.plans.physical.{KeyGroupedPartitioning, Par
import org.apache.spark.sql.catalyst.util.{truncatedString,
InternalRowComparableWrapper}
import org.apache.spark.sql.connector.catalog.Table
import org.apache.spark.sql.connector.read._
-import org.apache.spark.sql.execution.KeyGroupedPartitionedScan
+import org.apache.spark.sql.execution.{KeyGroupedPartitionedScan, SparkPlan}
import org.apache.spark.sql.execution.joins.StoragePartitionJoinParams
import org.apache.spark.util.ArrayImplicits._
@@ -44,7 +44,13 @@ case class BatchScanExec(
spjParams: StoragePartitionJoinParams = StoragePartitionJoinParams()
) extends DataSourceV2ScanExecBase with
KeyGroupedPartitionedScan[InputPartition] {
- @transient lazy val batch: Batch = if (scan == null) null else scan.toBatch
+ // What this node plans from `scan`, see [[PlannedScan]]. `makeCopy` and
`clone` hand it over to a
+ // copy over the same `Scan`. Any other new node plans its own, including
one the case-class
+ // `copy` makes.
+ @transient private var plannedScan: PlannedScan =
+ if (scan == null) null else new PlannedScan(scan)
+
+ def batch: Batch = if (scan == null) null else plannedScan.batch
// TODO: unify the equal/hashCode implementation for all data source v2
query plans.
override def equals(other: Any): Boolean = other match {
@@ -58,8 +64,7 @@ case class BatchScanExec(
override def hashCode(): Int = Objects.hash(batch, runtimeFilters)
- @transient override lazy val inputPartitions: Seq[InputPartition] =
- batch.planInputPartitions().toImmutableArraySeq
+ override def inputPartitions: Seq[InputPartition] =
plannedScan.inputPartitions
@transient private lazy val filteredPartitions: Seq[Seq[InputPartition]] = {
val dataSourceFilters = runtimeFilters.flatMap {
@@ -129,7 +134,7 @@ case class BatchScanExec(
}
}
- override lazy val readerFactory: PartitionReaderFactory =
batch.createReaderFactory()
+ override def readerFactory: PartitionReaderFactory =
plannedScan.readerFactory
override lazy val inputRDD: RDD[InternalRow] = {
val rdd = if (filteredPartitions.isEmpty && outputPartitioning ==
SinglePartition) {
@@ -152,6 +157,19 @@ case class BatchScanExec(
override def keyGroupedPartitioning: Option[Seq[Expression]] =
spjParams.keyGroupedPartitioning
+ override def makeCopy(newArgs: Array[AnyRef]): SparkPlan =
sharePlanning(super.makeCopy(newArgs))
+
+ // `TreeNode.clone` does not go through `makeCopy`.
+ override def clone(): SparkPlan = sharePlanning(super.clone())
+
+ private def sharePlanning(node: SparkPlan): SparkPlan = {
+ node match {
+ case c: BatchScanExec if c.scan eq scan => c.plannedScan = plannedScan
+ case _ =>
+ }
+ node
+ }
+
override def doCanonicalize(): BatchScanExec = {
this.copy(
output = output.map(QueryPlan.normalizeExpressions(_, output)),
@@ -171,3 +189,18 @@ case class BatchScanExec(
s"BatchScan ${table.name()}".trim
}
}
+
+/**
+ * A `Scan`, with the `Batch` and the input partitions a [[BatchScanExec]]
plans from it once, when
+ * a node first asks for them. The node hands it over to the copies `makeCopy`
and `clone` make of
+ * it, so they read the partitions the first node planned, with a reader
factory from the `Batch`
+ * that planned them. The partitions have to be the planned ones, because the
`Scan` can be shared
+ * with another node whose runtime filter prunes it first, e.g. the build side
of a DPP subquery
+ * (SPARK-60752). A pushed runtime filter replans them from a new `Batch`, but
keeps the planned
+ * `Batch`'s reader factory.
+ */
+private[v2] final class PlannedScan(val scan: Scan) {
+ lazy val batch: Batch = scan.toBatch
+ lazy val inputPartitions: Seq[InputPartition] =
batch.planInputPartitions().toImmutableArraySeq
+ lazy val readerFactory: PartitionReaderFactory = batch.createReaderFactory()
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index 4e112e84215b..74a6c2a7ca90 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -25,12 +25,13 @@ import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{Literal, TransformExpression}
import org.apache.spark.sql.catalyst.plans.physical
import org.apache.spark.sql.catalyst.plans.physical.KeyGroupedPartitioning
-import org.apache.spark.sql.connector.catalog.{Column, Identifier,
InMemoryTableCatalog}
+import org.apache.spark.sql.connector.catalog.{CatalogV2Implicits, Column,
Identifier, InMemoryTable, InMemoryTableCatalog,
InMemoryTableWithV2FilterCatalog, Table}
+import org.apache.spark.sql.connector.catalog.constraints.Constraint
import org.apache.spark.sql.connector.catalog.functions._
-import org.apache.spark.sql.connector.distributions.Distributions
+import org.apache.spark.sql.connector.distributions.{Distribution,
Distributions}
import org.apache.spark.sql.connector.expressions._
import org.apache.spark.sql.connector.expressions.Expressions._
-import org.apache.spark.sql.connector.read.HasPartitionKey
+import org.apache.spark.sql.connector.read.{HasPartitionKey, Scan, ScanBuilder}
import org.apache.spark.sql.execution.{RDDScanExec, SparkPlan}
import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
@@ -40,6 +41,7 @@ import org.apache.spark.sql.functions.{col, max}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf._
import org.apache.spark.sql.types._
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase {
private val functions = Seq(
@@ -514,6 +516,139 @@ class KeyGroupedPartitioningSuite extends
DistributionAndOrderingSuiteBase {
}
}
+ /**
+ * Runs `f` under the confs of a DPP subquery that reuses no broadcast, and
drops the tables of
+ * `catalog` after.
+ */
+ private def withDppSubqueryConfs(catalog: InMemoryTableCatalog)(f: => Unit):
Unit = {
+ withSQLConf(
+ SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+ try f finally catalog.clearTables()
+ }
+ }
+
+ /** Fills `items` and `purchases` with the ids 1 to 7, with `rowsPerKey`
rows in each purchase. */
+ private def insertDppSubqueryRows(ns: String, rowsPerKey: Int): Unit = {
+ sql(s"INSERT INTO $ns.$items VALUES " + (1 to 7).map(i =>
+ s"($i, 'x', 10.0, cast('2020-01-01' as timestamp))").mkString(", "))
+ sql(s"INSERT INTO $ns.$purchases VALUES " + (1 to 7).flatMap(i =>
+ Seq.fill(rowsPerKey)(s"($i, $i.0, cast('2020-01-01' as
timestamp))")).mkString(", "))
+ }
+
+ /**
+ * Checks the rows of `query`, and that its join plans no shuffle, with AQE
off unless `aqe` says
+ * otherwise. With AQE on, the copies of the join's scan plan their
partitions when the result
+ * stage is created and asks `supportsColumnar`, before the DPP subquery
prunes the shared `Scan`.
+ * That holds for a scan in the default `PARTITION_DEFINED` columnar support
mode.
+ *
+ * Each case runs with partition values pushed down and without. With them
pushed, this branch's
+ * scan pads its partitions to the common partition values of the join,
which hides a copy that
+ * planned fewer partitions, unless the build side's filter prunes every
split.
+ */
+ private def checkStoragePartitionedJoin(
+ query: String,
+ expected: Seq[Row],
+ aqe: Seq[Boolean] = Seq(false)): Unit = {
+ for (enabled <- aqe; pushPartValues <- Seq(true, false)) {
+ withSQLConf(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enabled.toString,
+ SQLConf.V2_BUCKETING_PUSH_PART_VALUES_ENABLED.key ->
pushPartValues.toString) {
+ val df = sql(query)
+ checkAnswer(df, expected)
+ val plan = df.queryExecution.executedPlan
+ assert(collectShuffles(plan).isEmpty,
+ s"The join should stay storage-partitioned:\n$plan")
+ }
+ }
+ }
+
+ test("SPARK-60752: a DPP subquery's build side does not change the layout of
the scan it " +
+ "shares with the join") {
+ // A DPP subquery that does not reuse a broadcast plans its build side
again, over the same
+ // `Scan` as the join's `purchases` scan. A scalar subquery filter on
`item_id` is a runtime
+ // filter of both, and the subquery runs it first, which prunes the shared
`Scan`. The join's
+ // scan is a copy made after planning, when the scalar subquery is
replaced by a reused one. Had
+ // it planned its partitions again, it would report fewer of them than
`items`, and the join
+ // would fail with "Can't zip RDDs". The V2 filter catalog's scans prune
on this predicate.
+ val catalogName = "v2_filter_cat"
+ withSQLConf(
+ s"spark.sql.catalog.$catalogName" ->
classOf[InMemoryTableWithV2FilterCatalog].getName) {
+ val v2FilterCatalog =
spark.sessionState.catalogManager.catalog(catalogName)
+ .asInstanceOf[InMemoryTableCatalog]
+ val ns = s"$catalogName.ns"
+ withDppSubqueryConfs(v2FilterCatalog) {
+ sql(s"CREATE TABLE $ns.$items (id BIGINT, name STRING, price FLOAT, " +
+ "arrive_time TIMESTAMP) PARTITIONED BY (id)")
+ sql(s"CREATE TABLE $ns.$purchases (item_id BIGINT, price FLOAT, time
TIMESTAMP) " +
+ "PARTITIONED BY (item_id)")
+ insertDppSubqueryRows(ns, rowsPerKey = 1)
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM $ns.$items i JOIN $ns.$purchases p " +
+ s"ON i.id = p.item_id WHERE i.id = " +
+ s"(SELECT max(item_id) FROM $ns.$purchases WHERE price > 4.0)",
+ Seq(Row(7L, 7.0f)))
+ }
+ }
+ }
+
+ test("SPARK-60752: a nested DPP filter on a DPP subquery's build side does
not change the " +
+ "layout of the scan it shares with the join") {
+ // As above, with a DPP filter from a broadcast dimension as the build
side's own runtime
+ // filter, which the in-memory scans prune on. Two rows per key give
`purchases` two splits per
+ // key, which its scan groups by key. With `flag = 2` the build side's
filter prunes every
+ // split.
+ withDppSubqueryConfs(catalog) {
+ createTable(items, itemsColumns, Array(identity("id")))
+ createTable(purchases, purchasesColumns, Array(identity("item_id")))
+ insertDppSubqueryRows("testcat.ns", rowsPerKey = 2)
+ createTable("dim", Array(Column.create("k", LongType),
Column.create("flag", IntegerType)),
+ Array.empty)
+ sql("INSERT INTO testcat.ns.dim VALUES " +
+ (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+ Seq(1 -> (5 to 7), 2 -> Nil).foreach { case (flag, ids) =>
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM testcat.ns.$items i JOIN " +
+ s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM
testcat.ns.$purchases p " +
+ s"JOIN testcat.ns.dim d ON p.item_id = d.k WHERE d.flag = $flag) p
" +
+ "ON i.id = p.item_id",
+ ids.flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))))
+ }
+ }
+ }
+
+ test("SPARK-60752: a DPP subquery's build side does not change the layout of
a shared scan " +
+ "that declares its columnar support mode, with AQE on") {
+ // A scan that declares its mode plans nothing when `supportsColumnar`
asks, so with AQE on the
+ // copies of the join's scan plan after the build side's filter too. This
is the nested shape
+ // above, over such a scan, with a dimension match, since an empty
broadcast lets AQE drop the
+ // join.
+ val catalogName = "unsupported_columnar_cat"
+ withSQLConf(
+ s"spark.sql.catalog.$catalogName" ->
classOf[UnsupportedColumnarModeCatalog].getName) {
+ val modeCatalog = spark.sessionState.catalogManager.catalog(catalogName)
+ .asInstanceOf[UnsupportedColumnarModeCatalog]
+ val ns = s"$catalogName.ns"
+ withDppSubqueryConfs(modeCatalog) {
+ createTable(items, itemsColumns, Array(identity("id")), catalog =
modeCatalog)
+ createTable(purchases, purchasesColumns, Array(identity("item_id")),
+ catalog = modeCatalog)
+ insertDppSubqueryRows(ns, rowsPerKey = 2)
+ createTable("dim", Array(Column.create("k", LongType),
Column.create("flag", IntegerType)),
+ Array.empty, catalog = modeCatalog)
+ sql(s"INSERT INTO $ns.dim VALUES " +
+ (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM $ns.$items i JOIN " +
+ s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM
$ns.$purchases p " +
+ s"JOIN $ns.dim d ON p.item_id = d.k WHERE d.flag = 1) p ON i.id =
p.item_id",
+ (5 to 7).flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))),
+ aqe = Seq(true, false))
+ }
+ }
+ }
+
test("partitioned join: join with two partition keys and matching & sorted
partitions") {
val items_partitions = Array(bucket(8, "id"), days("arrive_time"))
createTable(items, itemsColumns, items_partitions)
@@ -5430,3 +5565,47 @@ class KeyGroupedPartitioningSuite extends
DistributionAndOrderingSuiteBase {
}
}
+
+/**
+ * An in-memory catalog whose scans declare `UNSUPPORTED` as their columnar
support mode, so
+ * `supportsColumnar` answers without planning the scan's partitions.
+ */
+class UnsupportedColumnarModeCatalog extends InMemoryTableCatalog {
+ import CatalogV2Implicits._
+
+ // scalastyle:off argcount
+ override def createTable(
+ ident: Identifier,
+ columns: Array[Column],
+ partitions: Array[Transform],
+ properties: java.util.Map[String, String],
+ distribution: Distribution,
+ ordering: Array[SortOrder],
+ requiredNumPartitions: Option[Int],
+ advisoryPartitionSize: Option[Long],
+ constraints: Array[Constraint],
+ distributionStrictlyRequired: Boolean,
+ numRowsPerSplit: Int): Table = {
+ // scalastyle:on argcount
+ val table = new InMemoryTable(s"$name.${ident.quoted}", columns,
partitions, properties,
+ constraints, distribution, ordering, requiredNumPartitions,
advisoryPartitionSize,
+ distributionStrictlyRequired, numRowsPerSplit) {
+ override def newScanBuilder(options: CaseInsensitiveStringMap):
ScanBuilder =
+ new InMemoryScanBuilder(schema(), options) {
+ // This branch's `InMemoryScanBuilder` has no hook for the scan it
builds, so the built
+ // scan is copied into one that declares `UNSUPPORTED`.
+ override def build: Scan = {
+ val built = super.build.asInstanceOf[InMemoryBatchScan]
+ new InMemoryBatchScan(
+ built.data, built.readSchema, built.tableSchema,
built.options) {
+ override def columnarSupportMode(): Scan.ColumnarSupportMode =
+ Scan.ColumnarSupportMode.UNSUPPORTED
+ }
+ }
+ }
+ }
+ tables.put(ident, table)
+ namespaces.putIfAbsent(ident.namespace.toList, Map())
+ table
+ }
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
index aed11badb710..06ee4f0c554c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
@@ -23,10 +23,11 @@ import org.apache.spark.sql.QueryTest
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
import org.apache.spark.sql.catalyst.plans.logical.Deduplicate
+import org.apache.spark.sql.connector.read.{Batch, InputPartition,
PartitionReaderFactory, Scan}
import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.IntegerType
+import org.apache.spark.sql.types.{IntegerType, StructType}
import org.apache.spark.sql.vectorized.ColumnarBatch
class SparkPlanSuite extends QueryTest with SharedSparkSession {
@@ -146,6 +147,31 @@ class SparkPlanSuite extends QueryTest with
SharedSparkSession {
}
}
}
+
+ test("SPARK-60752: copies of a BatchScanExec share its Batch and the
partitions it planned") {
+ // A `Scan` that returns a new `Batch` from each `toBatch`, which a
connector may do.
+ class NewBatchScan extends Scan {
+ override def readSchema(): StructType = new StructType()
+ override def toBatch: Batch = new Batch {
+ override def planInputPartitions(): Array[InputPartition] = Array(new
InputPartition {})
+ override def createReaderFactory(): PartitionReaderFactory =
+ throw new UnsupportedOperationException()
+ }
+ }
+ val scan = BatchScanExec(Seq.empty, new NewBatchScan, Seq.empty, table =
null)
+ def sharesPlanning(copy: SparkPlan): Boolean = copy match {
+ case c: BatchScanExec =>
+ (c.batch eq scan.batch) && (c.inputPartitions eq scan.inputPartitions)
+ case _ => false
+ }
+ val args = scan.productIterator.map(_.asInstanceOf[AnyRef]).toArray
+
+ // Copies through `makeCopy` or `clone` share what the node planned, but
not over another
+ // `Scan`.
+ assert(sharesPlanning(scan.makeCopy(args)))
+ assert(sharesPlanning(scan.clone()))
+ assert(!sharesPlanning(scan.makeCopy(args.updated(1, new NewBatchScan))))
+ }
}
case class ColumnarOp(child: SparkPlan) extends UnaryExecNode {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]