This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new fbe3f9bc9d9b [SPARK-60752][4.3][SQL] Share a V2 scan's planned input
partitions between the copies of its BatchScanExec
fbe3f9bc9d9b is described below
commit fbe3f9bc9d9b11cd30bd1460a6052e3857280e68
Author: Peter Toth <[email protected]>
AuthorDate: Thu Nov 5 08:41:59 2026 -0800
[SPARK-60752][4.3][SQL] Share a V2 scan's planned input partitions between
the copies of its BatchScanExec
### What changes were proposed in this pull request?
Backport of #60030 to branch-4.3. The fix is the same, but its shape keeps
the signature of `BatchScanExec` in this maintenance branch:
- `BatchScanExec` keeps its `scan` field. Each node holds a `PlannedScan`
in a private field instead, built from `scan`.
- `makeCopy` and `clone` hand the node's `PlannedScan` over to a copy over
the same `Scan`. The rules that rewrite the node do that through `makeCopy`.
`clone` needs its own override, because `TreeNode.clone` does not go through
`makeCopy`.
- Any other new node plans its own, including one the case-class `copy`
makes. Nothing in Spark copies an executable `BatchScanExec` with `copy`.
- `PlannedScan` is package private here.
A `PlannedScan` holds the `Scan`, with the `Batch`, its reader factory and
the input partitions planned from it once, when a node first asks for them. So
the copies read the partitions the original node planned, with the reader
factory of the `Batch` that planned them. A pushed runtime filter still replans
from a new `Batch`, as before.
Nothing is planned earlier than before.
- The first node that asks plans the partitions, as `inputPartitions` did.
- A copy reuses them instead of planning again, so a connector's
`planInputPartitions()` runs less often.
Handing mutable state over in `makeCopy` and `clone` is what
`InMemoryRelation` does with its `statsOfPlanToCache`.
### Why are the changes needed?
A storage-partitioned join can fail when the build side of a DPP subquery
has a runtime filter of its own. With AQE off:
1. A DPP subquery that does not reuse a broadcast plans its build side
again. Its `BatchScanExec` wraps the same `Scan` instance as the join's scan on
that side.
2. The build side has a runtime filter of its own. It can be a DPP filter
from another join, or a scalar subquery filter routed there by SPARK-56467. The
DPP subquery runs first, and its `filter()` call prunes the shared `Scan`. The
runtime filtering contracts allow that.
3. The join's `BatchScanExec` is a copy made after planning.
`ReuseExchangeAndSubquery` made it when it replaced a subquery in the node's
runtime filters with a reused one. The copy plans its input partitions again,
from the already pruned `Scan`.
4. So the copy reports fewer partitions than the join was planned for, and
the join fails with `Can't zip RDDs with unequal numbers of partitions`.
For example, take a connector whose `filter()` drops the partitions an `=`
predicate rules out, with `items` partitioned by `id` and `purchases`
partitioned by `item_id`, ids 1 to 7 with one split per key. With DPP on,
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly=false`,
`spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio=10`,
`spark.sql.autoBroadcastJoinThreshold=-1` and AQE off:
```sql
SELECT i.id, i.name, p.price FROM items i JOIN purchases p ON i.id =
p.item_id
WHERE i.id = (SELECT max(item_id) FROM purchases WHERE price > 4.0)
```
fails with `Can't zip RDDs with unequal numbers of partitions: List(7, 1)`.
The same happens when `purchases` is first joined with a broadcast
dimension, whose DPP filter is then the build side's own runtime filter. A
connector that prunes on `IN` only is enough for that.
With AQE on, a scan in the default `PARTITION_DEFINED` columnar support
mode is not affected. AQE asks the copies `supportsColumnar` when it creates
the result stage, and in that mode this plans their partitions before the DPP
subquery runs. A scan that declares `SUPPORTED` or `UNSUPPORTED` plans nothing
there, so it fails the same way with AQE on.
### Does this PR introduce _any_ user-facing change?
Yes. Such a query now returns its rows instead of failing.
### How was this patch tested?
The tests of #60030, adapted to this branch. It has neither the Catalyst
runtime filter fixture nor the runtime filter test trait, so they are in
`KeyGroupedPartitioningSuite`. Each checks the rows and that the join plans no
shuffle.
- A scalar subquery filter, with one split per key, over
`InMemoryTableWithV2FilterCatalog`, whose scans prune on `=`. AQE off.
- A DPP filter from a broadcast dimension, with two splits per key, once
keeping 3 keys and once pruning every split. AQE off.
- The DPP filter case over a catalog whose scans declare `UNSUPPORTED`,
with AQE on and off.
Without the fix each fails with `Can't zip RDDs with unequal numbers of
partitions`, the last one with AQE on too.
A new test in `SparkPlanSuite`, over a `Scan` that returns a new `Batch`
from each `toBatch`. The copies `makeCopy` and `clone` make share the node's
`Batch` and partitions, and a `makeCopy` over another `Scan` does not. Without
the fix it fails at the first assertion.
Also ran `GroupPartitionsExecSuite`, `SameResultSuite`,
`SQLAppStatusListenerSuite`, `EnsureRequirementsSuite`, the
`DynamicPartitionPruningV2*` suites, `DataSourceV2SQLSuiteV2Filter`,
`DataSourceV2Suite`, `FileBasedDataSourceSuite`, `DataSourceV2OptionSuite`,
`PruneFileSourcePartitionsSuite`, `OrcV2SchemaPruningSuite`, the
`GroupBasedRowLevelOperation*` suites, `AvroRowReaderSuite` and `AvroV2Suite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 5.5
Closes #60125 from peter-toth/SPARK-60752-dpp-build-side-shared-scan-4.3.
Authored-by: Peter Toth <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../execution/datasources/v2/BatchScanExec.scala | 43 ++++-
.../connector/KeyGroupedPartitioningSuite.scala | 177 ++++++++++++++++++++-
.../spark/sql/execution/SparkPlanSuite.scala | 28 +++-
3 files changed, 240 insertions(+), 8 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
index 2e3394ac2a08..289d297cc240 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
@@ -29,7 +29,7 @@ import org.apache.spark.sql.connector.catalog.Table
import org.apache.spark.sql.connector.read._
import org.apache.spark.sql.connector.write.RowLevelOperation.Command.DELETE
import org.apache.spark.sql.connector.write.RowLevelOperationTable
-import org.apache.spark.sql.execution.EmptyRDDWithPartitions
+import org.apache.spark.sql.execution.{EmptyRDDWithPartitions, SparkPlan}
import org.apache.spark.sql.execution.metric.{SQLLastAttemptMetrics,
SQLMetric, SQLMetrics}
import org.apache.spark.util.ArrayImplicits._
@@ -45,7 +45,13 @@ case class BatchScanExec(
keyGroupedPartitioning: Option[Seq[Expression]] = None
) extends DataSourceV2ScanExecBase {
- @transient lazy val batch: Batch = if (scan == null) null else scan.toBatch
+ // What this node plans from `scan`, see [[PlannedScan]]. `makeCopy` and
`clone` hand it over to a
+ // copy over the same `Scan`. Any other new node plans its own, including
one the case-class
+ // `copy` makes.
+ @transient private var plannedScan: PlannedScan =
+ if (scan == null) null else new PlannedScan(scan)
+
+ def batch: Batch = if (scan == null) null else plannedScan.batch
override protected lazy val sparkMetrics: Map[String, SQLMetric] = {
val name = "number of output rows"
@@ -73,8 +79,7 @@ case class BatchScanExec(
override def hashCode(): Int = Objects.hash(batch, runtimeFilters,
keyGroupedPartitioning)
- @transient override lazy val inputPartitions: Seq[InputPartition] =
- batch.planInputPartitions().toImmutableArraySeq
+ override def inputPartitions: Seq[InputPartition] =
plannedScan.inputPartitions
// Visible for testing
@transient private[sql] lazy val filteredPartitions:
Seq[Option[InputPartition]] =
@@ -86,7 +91,7 @@ case class BatchScanExec(
outputPartitioning,
inputPartitions)
- override lazy val readerFactory: PartitionReaderFactory =
batch.createReaderFactory()
+ override def readerFactory: PartitionReaderFactory =
plannedScan.readerFactory
override lazy val inputRDD: RDD[InternalRow] = {
val rdd = if (filteredPartitions.isEmpty && outputPartitioning ==
SinglePartition) {
@@ -100,6 +105,19 @@ case class BatchScanExec(
rdd
}
+ override def makeCopy(newArgs: Array[AnyRef]): SparkPlan =
sharePlanning(super.makeCopy(newArgs))
+
+ // `TreeNode.clone` does not go through `makeCopy`.
+ override def clone(): SparkPlan = sharePlanning(super.clone())
+
+ private def sharePlanning(node: SparkPlan): SparkPlan = {
+ node match {
+ case c: BatchScanExec if c.scan eq scan => c.plannedScan = plannedScan
+ case _ =>
+ }
+ node
+ }
+
override def doCanonicalize(): BatchScanExec = {
this.copy(
output = output.map(QueryPlan.normalizeExpressions(_, output)),
@@ -121,3 +139,18 @@ case class BatchScanExec(
s"BatchScan ${table.name()}".trim
}
}
+
+/**
+ * A `Scan`, with the `Batch` and the input partitions a [[BatchScanExec]]
plans from it once, when
+ * a node first asks for them. The node hands it over to the copies `makeCopy`
and `clone` make of
+ * it, so they read the partitions the first node planned, with a reader
factory from the `Batch`
+ * that planned them. The partitions have to be the planned ones, because the
`Scan` can be shared
+ * with another node whose runtime filter prunes it first, e.g. the build side
of a DPP subquery
+ * (SPARK-60752). A pushed runtime filter replans them from a new `Batch`, but
keeps the planned
+ * `Batch`'s reader factory.
+ */
+private[v2] final class PlannedScan(val scan: Scan) {
+ lazy val batch: Batch = scan.toBatch
+ lazy val inputPartitions: Seq[InputPartition] =
batch.planInputPartitions().toImmutableArraySeq
+ lazy val readerFactory: PartitionReaderFactory = batch.createReaderFactory()
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index 4ff0d3142c51..2d34dcfffe15 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -27,11 +27,13 @@ import
org.apache.spark.sql.catalyst.expressions.{Ascending, AttributeReference,
import org.apache.spark.sql.catalyst.plans.Inner
import org.apache.spark.sql.catalyst.plans.physical
import org.apache.spark.sql.catalyst.plans.physical.KeyedPartitioning
-import org.apache.spark.sql.connector.catalog.{Column, Identifier,
InMemoryTableCatalog}
+import org.apache.spark.sql.connector.catalog.{CatalogV2Implicits, Column,
Identifier, InMemoryTable, InMemoryTableCatalog,
InMemoryTableWithV2FilterCatalog, Table}
+import org.apache.spark.sql.connector.catalog.constraints.Constraint
import org.apache.spark.sql.connector.catalog.functions._
-import org.apache.spark.sql.connector.distributions.Distributions
+import org.apache.spark.sql.connector.distributions.{Distribution,
Distributions}
import org.apache.spark.sql.connector.expressions._
import org.apache.spark.sql.connector.expressions.Expressions._
+import org.apache.spark.sql.connector.read.{InputPartition, Scan, ScanBuilder}
import org.apache.spark.sql.execution.{
ExtendedMode,
FormattedMode,
@@ -50,6 +52,7 @@ import org.apache.spark.sql.functions.{col, max}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf._
import org.apache.spark.sql.types._
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
class KeyGroupedPartitioningSuite extends DistributionAndOrderingSuiteBase
with ExplainSuiteHelper {
private val functions = Seq(
@@ -1510,6 +1513,133 @@ class KeyGroupedPartitioningSuite extends
DistributionAndOrderingSuiteBase with
}
}
+ /**
+ * Runs `f` under the confs of a DPP subquery that reuses no broadcast, and
drops the tables of
+ * `catalog` after.
+ */
+ private def withDppSubqueryConfs(catalog: InMemoryTableCatalog)(f: => Unit):
Unit = {
+ withSQLConf(
+ SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+ try f finally catalog.clearTables()
+ }
+ }
+
+ /** Fills `items` and `purchases` with the ids 1 to 7, with `rowsPerKey`
rows in each purchase. */
+ private def insertDppSubqueryRows(ns: String, rowsPerKey: Int): Unit = {
+ sql(s"INSERT INTO $ns.$items VALUES " + (1 to 7).map(i =>
+ s"($i, 'x', 10.0, cast('2020-01-01' as timestamp))").mkString(", "))
+ sql(s"INSERT INTO $ns.$purchases VALUES " + (1 to 7).flatMap(i =>
+ Seq.fill(rowsPerKey)(s"($i, $i.0, cast('2020-01-01' as
timestamp))")).mkString(", "))
+ }
+
+ /**
+ * Checks the rows of `query`, and that its join plans no shuffle, with AQE
off unless `aqe` says
+ * otherwise. With AQE on, the copies of the join's scan plan their
partitions when the result
+ * stage is created and asks `supportsColumnar`, before the DPP subquery
prunes the shared `Scan`.
+ * That holds for a scan in the default `PARTITION_DEFINED` columnar support
mode.
+ */
+ private def checkStoragePartitionedJoin(
+ query: String,
+ expected: Seq[Row],
+ aqe: Seq[Boolean] = Seq(false)): Unit = {
+ aqe.foreach { enabled =>
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> enabled.toString) {
+ val df = sql(query)
+ checkAnswer(df, expected)
+ val plan = df.queryExecution.executedPlan
+ assert(collectShuffles(plan).isEmpty,
+ s"The join should stay storage-partitioned:\n$plan")
+ }
+ }
+ }
+
+ test("SPARK-60752: a DPP subquery's build side does not change the layout of
the scan it " +
+ "shares with the join") {
+ // A DPP subquery that does not reuse a broadcast plans its build side
again, over the same
+ // `Scan` as the join's `purchases` scan. A scalar subquery filter on
`item_id` is a runtime
+ // filter of both, and the subquery runs it first, which prunes the shared
`Scan`. The join's
+ // scan is a copy made after planning, when the scalar subquery is
replaced by a reused one. Had
+ // it planned its partitions again, it would report fewer of them than
`items`, and the join
+ // would fail with "Can't zip RDDs". The V2 filter catalog's scans prune
on this predicate.
+ val catalogName = "v2_filter_cat"
+ withSQLConf(
+ s"spark.sql.catalog.$catalogName" ->
classOf[InMemoryTableWithV2FilterCatalog].getName) {
+ val v2FilterCatalog =
spark.sessionState.catalogManager.catalog(catalogName)
+ .asInstanceOf[InMemoryTableCatalog]
+ val ns = s"$catalogName.ns"
+ withDppSubqueryConfs(v2FilterCatalog) {
+ sql(s"CREATE TABLE $ns.$items (id BIGINT, name STRING, price FLOAT, " +
+ "arrive_time TIMESTAMP) PARTITIONED BY (id)")
+ sql(s"CREATE TABLE $ns.$purchases (item_id BIGINT, price FLOAT, time
TIMESTAMP) " +
+ "PARTITIONED BY (item_id)")
+ insertDppSubqueryRows(ns, rowsPerKey = 1)
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM $ns.$items i JOIN $ns.$purchases p " +
+ s"ON i.id = p.item_id WHERE i.id = " +
+ s"(SELECT max(item_id) FROM $ns.$purchases WHERE price > 4.0)",
+ Seq(Row(7L, 7.0f)))
+ }
+ }
+ }
+
+ test("SPARK-60752: a nested DPP filter on a DPP subquery's build side does
not change the " +
+ "layout of the scan it shares with the join") {
+ // As above, with a DPP filter from a broadcast dimension as the build
side's own runtime
+ // filter, which the in-memory scans prune on. Two rows per key give
`purchases` two splits per
+ // key, so a GroupPartitionsExec groups that side. With `flag = 2` the
build side's filter
+ // prunes every split.
+ withDppSubqueryConfs(catalog) {
+ createTable(items, itemsColumns, Array(identity("id")))
+ createTable(purchases, purchasesColumns, Array(identity("item_id")))
+ insertDppSubqueryRows("testcat.ns", rowsPerKey = 2)
+ createTable("dim", Array(Column.create("k", LongType),
Column.create("flag", IntegerType)),
+ Array.empty)
+ sql("INSERT INTO testcat.ns.dim VALUES " +
+ (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+ Seq(1 -> (5 to 7), 2 -> Nil).foreach { case (flag, ids) =>
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM testcat.ns.$items i JOIN " +
+ s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM
testcat.ns.$purchases p " +
+ s"JOIN testcat.ns.dim d ON p.item_id = d.k WHERE d.flag = $flag) p
" +
+ "ON i.id = p.item_id",
+ ids.flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))))
+ }
+ }
+ }
+
+ test("SPARK-60752: a DPP subquery's build side does not change the layout of
a shared scan " +
+ "that declares its columnar support mode, with AQE on") {
+ // A scan that declares its mode plans nothing when `supportsColumnar`
asks, so with AQE on the
+ // copies of the join's scan plan after the build side's filter too. This
is the nested shape
+ // above, over such a scan, with a dimension match, since an empty
broadcast lets AQE drop the
+ // join.
+ val catalogName = "unsupported_columnar_cat"
+ withSQLConf(
+ s"spark.sql.catalog.$catalogName" ->
classOf[UnsupportedColumnarModeCatalog].getName) {
+ val modeCatalog = spark.sessionState.catalogManager.catalog(catalogName)
+ .asInstanceOf[UnsupportedColumnarModeCatalog]
+ val ns = s"$catalogName.ns"
+ withDppSubqueryConfs(modeCatalog) {
+ createTable(items, itemsColumns, Array(identity("id")), catalog =
modeCatalog)
+ createTable(purchases, purchasesColumns, Array(identity("item_id")),
+ catalog = modeCatalog)
+ insertDppSubqueryRows(ns, rowsPerKey = 2)
+ createTable("dim", Array(Column.create("k", LongType),
Column.create("flag", IntegerType)),
+ Array.empty, catalog = modeCatalog)
+ sql(s"INSERT INTO $ns.dim VALUES " +
+ (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM $ns.$items i JOIN " +
+ s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM
$ns.$purchases p " +
+ s"JOIN $ns.dim d ON p.item_id = d.k WHERE d.flag = 1) p ON i.id =
p.item_id",
+ (5 to 7).flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))),
+ aqe = Seq(true, false))
+ }
+ }
+ }
+
test("partitioned join: join with two partition keys and matching & sorted
partitions") {
val items_partitions = Array(bucket(8, "id"), days("arrive_time"))
createTable(items, itemsColumns, items_partitions)
@@ -8217,3 +8347,46 @@ class KeyGroupedPartitioningSuite extends
DistributionAndOrderingSuiteBase with
}
}
+
+/**
+ * An in-memory catalog whose scans declare `UNSUPPORTED` as their columnar
support mode, so
+ * `supportsColumnar` answers without planning the scan's partitions.
+ */
+class UnsupportedColumnarModeCatalog extends InMemoryTableCatalog {
+ import CatalogV2Implicits._
+
+ // scalastyle:off argcount
+ override def createTable(
+ ident: Identifier,
+ columns: Array[Column],
+ partitions: Array[Transform],
+ properties: java.util.Map[String, String],
+ distribution: Distribution,
+ ordering: Array[SortOrder],
+ requiredNumPartitions: Option[Int],
+ advisoryPartitionSize: Option[Long],
+ constraints: Array[Constraint],
+ distributionStrictlyRequired: Boolean,
+ numRowsPerSplit: Int): Table = {
+ // scalastyle:on argcount
+ val table = new InMemoryTable(s"$name.${ident.quoted}", columns,
partitions, properties,
+ constraints, distribution, ordering, requiredNumPartitions,
advisoryPartitionSize,
+ distributionStrictlyRequired, numRowsPerSplit) {
+ override def newScanBuilder(options: CaseInsensitiveStringMap):
ScanBuilder =
+ new InMemoryScanBuilder(schema(), options) {
+ override protected def createScan(
+ partitions: Seq[InputPartition],
+ readSchema: StructType,
+ tableSchema: StructType,
+ options: CaseInsensitiveStringMap): BatchScanBaseClass =
+ new InMemoryBatchScan(partitions, readSchema, tableSchema,
options) {
+ override def columnarSupportMode(): Scan.ColumnarSupportMode =
+ Scan.ColumnarSupportMode.UNSUPPORTED
+ }
+ }
+ }
+ tables.put(ident, table)
+ namespaces.putIfAbsent(ident.namespace.toList, Map())
+ table
+ }
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
index c0f5ad61d72e..dfee04b47173 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
@@ -31,10 +31,11 @@ import org.apache.spark.sql.catalyst.expressions.{
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
import org.apache.spark.sql.catalyst.plans.logical.Deduplicate
import org.apache.spark.sql.catalyst.trees.LeafLike
+import org.apache.spark.sql.connector.read.{Batch, InputPartition,
PartitionReaderFactory, Scan}
import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{DataType, IntegerType, StringType}
+import org.apache.spark.sql.types.{DataType, IntegerType, StringType,
StructType}
import org.apache.spark.sql.vectorized.ColumnarBatch
import org.apache.spark.util.ThreadUtils
@@ -256,6 +257,31 @@ class SparkPlanSuite extends SharedSparkSession {
assert(canonicalizedTypes == Seq(IntegerType, StringType),
s"Expected [IntegerType, StringType] but got $canonicalizedTypes")
}
+
+ test("SPARK-60752: copies of a BatchScanExec share its Batch and the
partitions it planned") {
+ // A `Scan` that returns a new `Batch` from each `toBatch`, which a
connector may do.
+ class NewBatchScan extends Scan {
+ override def readSchema(): StructType = new StructType()
+ override def toBatch: Batch = new Batch {
+ override def planInputPartitions(): Array[InputPartition] = Array(new
InputPartition {})
+ override def createReaderFactory(): PartitionReaderFactory =
+ throw new UnsupportedOperationException()
+ }
+ }
+ val scan = BatchScanExec(Seq.empty, new NewBatchScan, Seq.empty, table =
null)
+ def sharesPlanning(copy: SparkPlan): Boolean = copy match {
+ case c: BatchScanExec =>
+ (c.batch eq scan.batch) && (c.inputPartitions eq scan.inputPartitions)
+ case _ => false
+ }
+ val args = scan.productIterator.map(_.asInstanceOf[AnyRef]).toArray
+
+ // Copies through `makeCopy` or `clone` share what the node planned, but
not over another
+ // `Scan`.
+ assert(sharesPlanning(scan.makeCopy(args)))
+ assert(sharesPlanning(scan.clone()))
+ assert(!sharesPlanning(scan.makeCopy(args.updated(1, new NewBatchScan))))
+ }
}
case class ColumnarOp(child: SparkPlan) extends UnaryExecNode {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]