This is an automated email from the ASF dual-hosted git repository.
peter-toth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 71316c632b05 [SPARK-60752][SQL] Share a V2 scan's planned input
partitions between the copies of its BatchScanExec
71316c632b05 is described below
commit 71316c632b05d1860e6450c154b2076e27322432
Author: Peter Toth <[email protected]>
AuthorDate: Thu Nov 5 09:27:31 2026 +0100
[SPARK-60752][SQL] Share a V2 scan's planned input partitions between the
copies of its BatchScanExec
### What changes were proposed in this pull request?
`BatchScanExec` now takes a `PlannedScan` in place of its `Scan`. A
`PlannedScan` holds the `Scan`, with the `Batch`, its reader factory and the
input partitions planned from it once, when a node first asks for them.
`DataSourceV2Strategy` builds one for each node, and the copies later rules
make of the node share it. So they read the partitions the original node
planned, with the reader factory of the `Batch` that planned them. A pushed
runtime filter still replans from a new `Batch` [...]
Nothing is planned earlier than before.
- The first node that asks plans the partitions, as `inputPartitions` did.
- A copy reuses them instead of planning again, so a connector's
`planInputPartitions()` runs less often.
This follows SPARK-59289, which made the derived values of
`GroupPartitionsExec` constructor fields for the same reason. A lazy val
belongs to one instance, and a copy computes it again. Sharing a holder between
copies is what `InMemoryRelation` does with its `CachedRDDBuilder`.
### Why are the changes needed?
A storage-partitioned join can fail when the build side of a DPP subquery
has a runtime filter of its own. With AQE off:
1. A DPP subquery that does not reuse a broadcast plans its build side
again. Its `BatchScanExec` wraps the same `Scan` instance as the join's scan on
that side.
2. The build side has a runtime filter of its own. It can be a DPP filter
from another join, or a scalar subquery filter routed there by SPARK-56467. The
DPP subquery runs first, and its `filter()` call prunes the shared `Scan`. The
runtime filtering contracts allow that.
3. The join's `BatchScanExec` is a copy made after planning.
`ReuseExchangeAndSubquery` made it when it replaced a subquery in the node's
runtime filters with a reused one. The copy plans its input partitions again,
from the already pruned `Scan`.
4. So the copy reports fewer partitions than the join was planned for:
- without a `GroupPartitionsExec` on that side, the join fails with
`Can't zip RDDs with unequal numbers of partitions`;
- with one, it fails with `[INTERNAL_ERROR] GroupPartitionsExec's child
no longer reports the partitioning it was planned over`.
For example, take a connector whose `filter()` drops the partitions an `=`
predicate rules out, with `items` partitioned by `id` and `purchases`
partitioned by `item_id`, ids 1 to 7 with one split per key. With DPP on,
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly=false`,
`spark.sql.optimizer.dynamicPartitionPruning.fallbackFilterRatio=10`,
`spark.sql.autoBroadcastJoinThreshold=-1` and AQE off:
```sql
SELECT i.id, i.name, p.price FROM items i JOIN purchases p ON i.id =
p.item_id
WHERE i.id = (SELECT max(item_id) FROM purchases WHERE price > 4.0)
```
fails with `Can't zip RDDs with unequal numbers of partitions: List(7, 1)`.
The same happens when `purchases` is first joined with a broadcast
dimension, whose DPP filter is then the build side's own runtime filter. A
connector that prunes on `IN` only is enough for that.
With AQE on, a scan in the default `PARTITION_DEFINED` columnar support
mode is not affected. AQE asks the copies `supportsColumnar` when it creates
the result stage, and in that mode this plans their partitions before the DPP
subquery runs. A scan that declares `SUPPORTED` or `UNSUPPORTED` plans nothing
there, so it fails the same way with AQE on.
Affected releases:
- 4.2.0 and later (branch-4.2, branch-4.3, branch-4.x and master), as
above. The `GroupPartitionsExec` internal error comes from SPARK-59289, which
only branch-4.x and master have.
- 4.1 and 4.0, with AQE off, only with
`spark.sql.sources.v2.bucketing.pushPartValues.enabled=false` or when the build
side's filter prunes every split. Otherwise their `BatchScanExec` pads to the
partition values in `spjParams`, which a copy keeps.
This PR is for master. branch-4.x will take the same change. The backports
to branch-4.3 and branch-4.2 will keep the `scan` field and hand the
`PlannedScan` over in `makeCopy` and `clone` instead, so the signature of
`BatchScanExec` stays. branch-4.1 and branch-4.0 need a different fix.
### Does this PR introduce _any_ user-facing change?
Yes. Such a query now returns its rows instead of failing.
For code that builds or rebuilds `BatchScanExec`, the second parameter is
now a `PlannedScan`. A new node takes `new PlannedScan(scan)`. A rule that
rebuilds a node over the same `Scan` passes on its `plannedScan`, or uses
`copy`, so the rebuilt node keeps the planned partitions. Code that matches the
node reads the scan through `BatchScanExec.scan`, or with `case
BatchScanExec(_, PlannedScan(scan), ...)`.
### How was this patch tested?
New tests in `KeyGroupedPartitioningRuntimeFilterTests`, with AQE off. Each
checks the rows and that the join plans no shuffle. With AQE on, the fixtures'
scans keep the default columnar support mode, so those runs cannot fail without
the fix.
- A scalar subquery filter, with one split per key. Without the fix it
fails in `KeyGroupedPartitioningCatalystRuntimeFilterSuite` with `List(7, 1)`.
`KeyGroupedPartitioningRuntimeFilterSuite` runs it too, but its fixture prunes
on `IN` only, so it passes there without the fix.
- A DPP filter from a broadcast dimension, with two splits per key, once
keeping 3 keys and once pruning every split. Without the fix each fails in both
suites with the `GroupPartitionsExec` internal error.
A new test in `KeyGroupedPartitioningSuite` runs the DPP filter case over a
scan that declares `UNSUPPORTED`, with AQE on and off. Without the fix it fails
with the `GroupPartitionsExec` internal error with AQE on too.
A new test in `SparkPlanSuite`, over a `Scan` that returns a new `Batch`
from each `toBatch`. The copies `makeCopy`, `clone` and `copy` make share the
node's `Batch` and partitions. Without the fix it fails at the first assertion.
The SPARK-60618 test that rebuilds a scan with new partitions now gives the
rebuilt node a new `PlannedScan`, since a copy keeps the planned ones.
Also ran `KeyGroupedPartitioningSuite`, `GroupPartitionsExecSuite`,
`SameResultSuite`, `SQLAppStatusListenerSuite`, `EnsureRequirementsSuite`, the
`DynamicPartitionPruningV2*` suites, `DataSourceV2CatalystRuntimeFilterSuite`,
`DataSourceV2SQLSuiteV2Filter`, `DSv2PlanMergingSuite`, and the suites whose
`BatchScanExec` patterns now match the `Scan` through `PlannedScan`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 5.5
Closes #60030 from peter-toth/SPARK-60752-dpp-build-side-shared-scan.
Authored-by: Peter Toth <[email protected]>
Signed-off-by: Peter Toth <[email protected]>
---
.../apache/spark/sql/avro/AvroRowReaderSuite.scala | 4 +-
.../org/apache/spark/sql/avro/AvroSuite.scala | 8 +-
.../execution/datasources/v2/BatchScanExec.scala | 35 +++-
.../datasources/v2/DataSourceV2Strategy.scala | 4 +-
.../datasources/v2/GroupPartitionsExec.scala | 6 +-
.../spark/sql/FileBasedDataSourceSuite.scala | 6 +-
.../sql/connector/DataSourceV2OptionSuite.scala | 6 +-
.../connector/KeyGroupedPartitioningSuite.scala | 183 ++++++++++++++++++++-
.../spark/sql/execution/SparkPlanSuite.scala | 34 +++-
.../PruneFileSourcePartitionsSuite.scala | 4 +-
.../datasources/PrunePartitionSuiteBase.scala | 5 +-
.../datasources/orc/OrcV2SchemaPruningSuite.scala | 4 +-
.../execution/ui/SQLAppStatusListenerSuite.scala | 10 +-
13 files changed, 266 insertions(+), 43 deletions(-)
diff --git
a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroRowReaderSuite.scala
b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroRowReaderSuite.scala
index c8dabde8004d..99ce65469f53 100644
---
a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroRowReaderSuite.scala
+++
b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroRowReaderSuite.scala
@@ -29,7 +29,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
import org.apache.spark.sql.catalyst.{InternalRow, NoopFilters}
import org.apache.spark.sql.catalyst.util.RebaseDateTime.RebaseSpec
-import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
PlannedScan}
import org.apache.spark.sql.internal.LegacyBehaviorPolicy._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
@@ -56,7 +56,7 @@ class AvroRowReaderSuite extends SharedSparkSession {
val df = spark.read.format("avro").load(dir.getCanonicalPath)
val fileScan = df.queryExecution.executedPlan collectFirst {
- case BatchScanExec(_, f: AvroScan, _, _, _, _, _, _) => f
+ case BatchScanExec(_, PlannedScan(f: AvroScan), _, _, _, _, _, _) => f
}
val filePath = fileScan.get.fileIndex.inputFiles(0)
val fileSize = new File(new URI(filePath)).length
diff --git
a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index 6296a32336a4..a4076247acb6 100644
--- a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -45,7 +45,7 @@ import
org.apache.spark.sql.catalyst.util.DateTimeTestUtils.{withDefaultTimeZone
import org.apache.spark.sql.connector.catalog.TableCapability
import org.apache.spark.sql.execution.{FileSourceScanExec, FormattedMode,
SparkPlan}
import org.apache.spark.sql.execution.datasources.{CommonFileDataSourceSuite,
DataSource, FilePartition}
-import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
DataSourceV2ScanRelation, FileDataSourceV2, FileTable}
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
DataSourceV2ScanRelation, FileDataSourceV2, FileTable, PlannedScan}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.LegacyBehaviorPolicy
import org.apache.spark.sql.internal.LegacyBehaviorPolicy._
@@ -4210,7 +4210,7 @@ class AvroV2Suite extends AvroSuite with
ExplainSuiteHelper {
})
val fileScan = df.queryExecution.executedPlan collectFirst {
- case BatchScanExec(_, f: AvroScan, _, _, _, _, _, _) => f
+ case BatchScanExec(_, PlannedScan(f: AvroScan), _, _, _, _, _, _) => f
}
assert(fileScan.nonEmpty)
assert(fileScan.get.partitionFilters.nonEmpty)
@@ -4244,7 +4244,7 @@ class AvroV2Suite extends AvroSuite with
ExplainSuiteHelper {
assert(filterCondition.isDefined)
val fileScan = df.queryExecution.executedPlan collectFirst {
- case BatchScanExec(_, f: AvroScan, _, _, _, _, _, _) => f
+ case BatchScanExec(_, PlannedScan(f: AvroScan), _, _, _, _, _, _) => f
}
assert(fileScan.nonEmpty)
assert(fileScan.get.partitionFilters.isEmpty)
@@ -4325,7 +4325,7 @@ class AvroV2Suite extends AvroSuite with
ExplainSuiteHelper {
.where("value = 'a'")
val fileScan = df.queryExecution.executedPlan collectFirst {
- case BatchScanExec(_, f: AvroScan, _, _, _, _, _, _) => f
+ case BatchScanExec(_, PlannedScan(f: AvroScan), _, _, _, _, _, _)
=> f
}
assert(fileScan.nonEmpty)
if (filtersPushdown) {
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
index 0f4fe2710d55..a81431d7bd83 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
@@ -39,6 +39,8 @@ import org.apache.spark.util.ArrayImplicits._
/**
* Physical plan node for scanning a batch of data from a data source v2.
*
+ * @param plannedScan The `Scan` to read, with the `Batch` and the input
partitions planned from it,
+ * see [[PlannedScan]].
* @param storageFilters Predicates the scan's readers may evaluate, to skip
reading the other
* columns of the rows they reject, see
[[SupportsStorageFilters]]. Non-empty
* only for a scan that declared their columns. Each
task's readers get them
@@ -49,7 +51,7 @@ import org.apache.spark.util.ArrayImplicits._
*/
case class BatchScanExec(
output: Seq[AttributeReference],
- @transient scan: Scan,
+ @transient plannedScan: PlannedScan,
runtimeFilters: Seq[Expression],
ordering: Option[Seq[SortOrder]] = None,
@transient table: Table,
@@ -67,7 +69,9 @@ case class BatchScanExec(
private def scanOutput: Seq[AttributeReference] =
if (appendsCheckedColumn) output.init else output
- @transient lazy val batch: Batch = if (scan == null) null else scan.toBatch
+ override def scan: Scan = if (plannedScan == null) null else plannedScan.scan
+
+ def batch: Batch = if (scan == null) null else plannedScan.batch
override protected lazy val sparkMetrics: Map[String, SQLMetric] = {
val name = "number of output rows"
@@ -108,8 +112,7 @@ case class BatchScanExec(
@transient lazy val prunedKeyGroupedPartitioning: Option[Seq[Expression]] =
keyGroupedPartitioning.map(_.filter(_.references.subsetOf(outputSet)))
- @transient override lazy val inputPartitions: Seq[InputPartition] =
- batch.planInputPartitions().toImmutableArraySeq
+ override def inputPartitions: Seq[InputPartition] =
plannedScan.inputPartitions
// Visible for testing. Runtime filtering resolves the scan's columns by
name, which the checked
// column must not take part in.
@@ -122,7 +125,7 @@ case class BatchScanExec(
reportedKeyedPartitioning,
inputPartitions)
- override lazy val readerFactory: PartitionReaderFactory =
batch.createReaderFactory()
+ override def readerFactory: PartitionReaderFactory =
plannedScan.readerFactory
// What every task's readers get, the storage filters ANDed into one. A
runtime bloom arrives as
// a ScalarSubquery over the build side, which SparkPlan has computed by the
time this is forced.
@@ -181,3 +184,25 @@ case class BatchScanExec(
s"BatchScan ${table.name()}".trim
}
}
+
+/**
+ * A `Scan`, with the `Batch` and the input partitions a [[BatchScanExec]]
plans from it once, when
+ * a node first asks for them. The copies later rules make of the node share
it, so they read the
+ * partitions the first node planned, with a reader factory from the `Batch`
that planned them.
+ * The partitions have to be the planned ones, because the `Scan` can be
shared with another node
+ * whose runtime filter prunes it first, e.g. the build side of a DPP subquery
(SPARK-60752). A
+ * pushed runtime filter replans them from a new `Batch`, but keeps the
planned `Batch`'s reader
+ * factory.
+ *
+ * A rule that rebuilds a node over the same `Scan` passes on its
`plannedScan`, or uses `copy`. A
+ * new `PlannedScan` is for a node that is planned again, or one over another
`Scan`.
+ */
+final class PlannedScan(val scan: Scan) {
+ lazy val batch: Batch = scan.toBatch
+ lazy val inputPartitions: Seq[InputPartition] =
batch.planInputPartitions().toImmutableArraySeq
+ lazy val readerFactory: PartitionReaderFactory = batch.createReaderFactory()
+}
+
+object PlannedScan {
+ def unapply(plannedScan: PlannedScan): Some[Scan] = Some(plannedScan.scan)
+}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
index 09b9dee9ceec..deb50d797f4c 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
@@ -258,8 +258,8 @@ class DataSourceV2Strategy(session: SparkSession) extends
Strategy with Predicat
val storageFilters = storageFiltersFor(relation, filtersAbove)
val checked = StorageFilterPlanning.checkedColumn(
session.sessionState.conf, storageFilters, relation.output)
- val batchExec = BatchScanExec(relation.output ++ checked, relation.scan,
runtimeFilters,
- relation.ordering, relation.relation.table, keyGroupedPartitioning,
+ val batchExec = BatchScanExec(relation.output ++ checked, new
PlannedScan(relation.scan),
+ runtimeFilters, relation.ordering, relation.relation.table,
keyGroupedPartitioning,
storageFilters, checked.isDefined)
// With the checked column the scan's output differs from `project`, so
a Project drops it.
DataSourceV2Strategy.withProjectAndFilter(project,
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
index f04d4035b532..b9bf4e19f52a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala
@@ -128,9 +128,9 @@ case class GroupPartitionsExec(
* The comparison is by reference for a child that reports the same object,
which the columnar and
* codegen wrappers do. A rule that rewrites a subquery in the runtime or
storage filters of a
* `BatchScanExec` after `EnsureRequirements` rebuilds it, so there it
compares the partition keys
- * by value, and reads equal only while the connector returns the same input
partitions for the
- * same scan. Such rules are `PlanAdaptiveDynamicPruningFilters` and
`ReuseAdaptiveSubquery` with
- * AQE, and `ReuseExchangeAndSubquery` without it.
+ * by value. Such rules are `PlanAdaptiveDynamicPruningFilters` and
`ReuseAdaptiveSubquery` with
+ * AQE, and `ReuseExchangeAndSubquery` without it. The rebuilt node shares
the input partitions
+ * the first one planned (`PlannedScan`), so its keys come from the same
partitions.
*
* Asked on the read rather than in `withNewChildInternal` because
canonicalization rebuilds this
* node over a canonicalized child whose partitioning cannot be read at all:
a canonicalized
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
index d5c6a9676063..4d176a2e275b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
@@ -38,7 +38,7 @@ import
org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPr
import org.apache.spark.sql.execution.{FileSourceScanLike, SimpleMode}
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
import org.apache.spark.sql.execution.datasources.FilePartition
-import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, FileScan}
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, FileScan,
PlannedScan}
import org.apache.spark.sql.execution.datasources.v2.orc.OrcScan
import org.apache.spark.sql.execution.datasources.v2.parquet.ParquetScan
import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec,
SortMergeJoinExec}
@@ -1038,7 +1038,7 @@ class FileBasedDataSourceSuite extends SharedSparkSession
})
val fileScan = df.queryExecution.executedPlan collectFirst {
- case BatchScanExec(_, f: FileScan, _, _, _, _, _, _) => f
+ case BatchScanExec(_, PlannedScan(f: FileScan), _, _, _, _, _, _)
=> f
}
assert(fileScan.nonEmpty)
assert(fileScan.get.partitionFilters.nonEmpty)
@@ -1079,7 +1079,7 @@ class FileBasedDataSourceSuite extends SharedSparkSession
assert(filterCondition.isDefined)
val fileScan = df.queryExecution.executedPlan collectFirst {
- case BatchScanExec(_, f: FileScan, _, _, _, _, _, _) => f
+ case BatchScanExec(_, PlannedScan(f: FileScan), _, _, _, _, _, _)
=> f
}
assert(fileScan.nonEmpty)
assert(fileScan.get.partitionFilters.isEmpty)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
index 15e65ce7b852..4ac50b13e1ce 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
@@ -346,7 +346,8 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase {
checkAnswer(df, Seq(Row(1, "a"), Row(2, "b")))
collected = df.queryExecution.executedPlan.collect {
- case BatchScanExec(_, scan: InMemoryBaseTable#InMemoryBatchScan, _, _,
_, _, _, _) =>
+ case BatchScanExec(
+ _, PlannedScan(scan: InMemoryBaseTable#InMemoryBatchScan), _, _,
_, _, _, _) =>
assert(scan.options.get("split-size") === "5")
}
assert (collected.size == 1)
@@ -381,7 +382,8 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase {
checkAnswer(df, Seq(Row(1, "a"), Row(2, "b")))
collected = df.queryExecution.executedPlan.collect {
- case BatchScanExec(_, scan: InMemoryBaseTable#InMemoryBatchScan, _, _,
_, _, _, _) =>
+ case BatchScanExec(
+ _, PlannedScan(scan: InMemoryBaseTable#InMemoryBatchScan), _, _,
_, _, _, _) =>
assert(scan.options.get("split-size") === "5")
}
assert (collected.size == 1)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
index c18a7bcec8a9..182b2f8d0df2 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala
@@ -36,7 +36,7 @@ import org.apache.spark.sql.connector.catalog.functions._
import org.apache.spark.sql.connector.distributions.{Distribution,
Distributions}
import org.apache.spark.sql.connector.expressions._
import org.apache.spark.sql.connector.expressions.Expressions._
-import org.apache.spark.sql.connector.read.{Batch, Scan, ScanBuilder,
SupportsReportOrdering, SupportsReportPartitioning}
+import org.apache.spark.sql.connector.read.{Batch, InputPartition, Scan,
ScanBuilder, SupportsReportOrdering, SupportsReportPartitioning}
import
org.apache.spark.sql.connector.read.partitioning.{KeyGroupedPartitioning,
Partitioning}
import org.apache.spark.sql.execution.{
ExtendedMode,
@@ -52,7 +52,7 @@ import org.apache.spark.sql.execution.{
WholeStageCodegenExec}
import org.apache.spark.sql.execution.adaptive.{AQEShuffleReadExec,
ResultQueryStageExec, ShuffleQueryStageExec}
import org.apache.spark.sql.execution.aggregate.{BaseAggregateExec,
HashAggregateExec, SortAggregateExec}
-import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
DataSourceV2ScanRelation, GroupPartitionsExec}
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
DataSourceV2ScanRelation, GroupPartitionsExec, PlannedScan}
import org.apache.spark.sql.execution.exchange.{EnsureRequirements,
ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike,
ValidateRequirements}
import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec,
ShuffledHashJoinExec, ShuffledJoin, SortMergeJoinExec}
import org.apache.spark.sql.execution.metric.SQLMetricsTestUtils
@@ -490,8 +490,8 @@ trait KeyGroupedPartitioningRuntimeFilterTests extends
KeyGroupedPartitioningSui
// Shape 4: a scalar subquery over the keyed column is a runtime filter
of the keyed scan too.
// Without this fix it returns 1 of 3 rows with AQE off, or fails to zip
with the Catalyst
- // runtime filters. DPP stays off here; with it on and AQE off the
Catalyst runtime filters
- // hit a separate, pre-existing failure (SPARK-60752).
+ // runtime filters. DPP stays off here, so the scalar subquery is the
keyed scan's only
+ // runtime filter. The SPARK-60752 tests below cover a scalar subquery
filter with DPP on.
Seq(true, false).foreach { aqe =>
withSQLConf(
SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "false",
@@ -511,6 +511,84 @@ trait KeyGroupedPartitioningRuntimeFilterTests extends
KeyGroupedPartitioningSui
}
}
}
+
+ /**
+ * Runs `f` over `items` and `purchases`, both partitioned by the ids 1 to
7, with `rowsPerKey`
+ * rows in each `purchases` key, under the confs of a DPP subquery that
reuses no broadcast.
+ */
+ private def withDppSubqueryTables(rowsPerKey: Int)(f: => Unit): Unit = {
+ withSQLConf(
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+ createTable(items, itemsColumns, Array(identity("id")))
+ sql(s"INSERT INTO testcat.ns.$items VALUES " + (1 to 7).map(i =>
+ s"($i, 'x', 10.0, cast('2020-01-01' as timestamp))").mkString(", "))
+ createTable(purchases, purchasesColumns, Array(identity("item_id")))
+ sql(s"INSERT INTO testcat.ns.$purchases VALUES " + (1 to 7).flatMap(i =>
+ Seq.fill(rowsPerKey)(s"($i, $i.0, cast('2020-01-01' as
timestamp))")).mkString(", "))
+ f
+ }
+ }
+
+ /**
+ * Checks the rows of `query`, and that its join plans no shuffle. With AQE
off only. With AQE on,
+ * the copies of the join's scan plan their partitions when the result stage
is created and asks
+ * `supportsColumnar`, before the DPP subquery prunes the shared `Scan`.
That holds for a scan in
+ * the default `PARTITION_DEFINED` columnar support mode, as the fixtures'
scans are.
+ * `KeyGroupedPartitioningSuite` runs AQE on over a scan that declares its
mode.
+ */
+ private def checkStoragePartitionedJoin(query: String, expected: Seq[Row]):
Unit = {
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+ val df = sql(query)
+ checkAnswer(df, expected)
+ val plan = df.queryExecution.executedPlan
+ assert(collectShuffles(plan).isEmpty,
+ s"The join should stay storage-partitioned:\n$plan")
+ }
+ }
+
+ test("SPARK-60752: a DPP subquery's build side does not change the layout of
the scan it " +
+ "shares with the join") {
+ // A DPP subquery that does not reuse a broadcast plans its build side
again, over the same
+ // `Scan` as the join's `purchases` scan. A scalar subquery filter on
`item_id` is a runtime
+ // filter of both, and the subquery runs it first, which prunes the shared
`Scan`. The join's
+ // scan is a copy made after planning, when the scalar subquery is
replaced by a reused one. Had
+ // it planned its partitions again, it would report fewer of them than
`items`, and the join
+ // would fail with "Can't zip RDDs". Of the two suites that run this, only
the Catalyst one
+ // prunes on this predicate.
+ withDppSubqueryTables(rowsPerKey = 1) {
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM testcat.ns.$items i JOIN
testcat.ns.$purchases p " +
+ s"ON i.id = p.item_id WHERE i.id = " +
+ s"(SELECT max(item_id) FROM testcat.ns.$purchases WHERE price >
4.0)",
+ Seq(Row(7L, 7.0f)))
+ }
+ }
+
+ test("SPARK-60752: a nested DPP filter on a DPP subquery's build side does
not change the " +
+ "layout of the scan it shares with the join") {
+ // As above, with a DPP filter from a broadcast dimension as the build
side's own runtime
+ // filter, which every fixture prunes on. Two rows per key give
`purchases` two splits per key,
+ // so a GroupPartitionsExec groups that side and checks the layout of the
copied scan. With
+ // `flag = 2` the build side's filter prunes every split.
+ val dim = "dim"
+ withDppSubqueryTables(rowsPerKey = 2) {
+ createTable(dim, Array(Column.create("k", LongType),
Column.create("flag", IntegerType)),
+ Array.empty)
+ sql(s"INSERT INTO testcat.ns.$dim VALUES " +
+ (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+ Seq(1 -> (5 to 7), 2 -> Nil).foreach { case (flag, ids) =>
+ checkStoragePartitionedJoin(
+ s"SELECT i.id, p.price FROM testcat.ns.$items i JOIN " +
+ s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM
testcat.ns.$purchases p " +
+ s"JOIN testcat.ns.$dim d ON p.item_id = d.k WHERE d.flag = $flag)
p " +
+ "ON i.id = p.item_id",
+ ids.flatMap(i => Seq.fill(2)(Row(i.toLong, i.toFloat))))
+ }
+ }
+ }
}
@ExtendedSQLTest
@@ -2256,10 +2334,12 @@ class KeyGroupedPartitioningSuite
}
}
try {
- // A rewritten runtime filter makes the scan copy distinct for tree
replacement, while
- // the true literal leaves the connector's replacement partitions
unchanged.
+ // A new `PlannedScan` makes the copy plan the connector's
replacement partitions, and a
+ // rewritten runtime filter makes it distinct for tree replacement,
while the true literal
+ // leaves those partitions unchanged.
inMemoryScan.data = newData
- val rebuilt = leftScan.copy(runtimeFilters =
Seq(Literal.TrueLiteral))
+ val rebuilt = leftScan.copy(plannedScan = new
PlannedScan(leftScan.scan),
+ runtimeFilters = Seq(Literal.TrueLiteral))
assert(rebuilt.outputPartitioning.numPartitions ==
replacementKeys.size)
assert(rebuilt.outputPartitioning != leftScan.outputPartitioning ||
replacementKeys == Seq(1, 2))
@@ -2312,6 +2392,55 @@ class KeyGroupedPartitioningSuite
}
}
+ test("SPARK-60752: a DPP subquery's build side does not change the layout of
a shared scan " +
+ "that declares its columnar support mode, with AQE on") {
+ // With AQE on, the copies of a scan in the default `PARTITION_DEFINED`
mode plan their
+ // partitions when the result stage asks `supportsColumnar`, before the
DPP subquery prunes the
+ // shared `Scan`. A scan that declares its mode skips that, so the copies
plan after the build
+ // side's filter, as with AQE off. This is the nested DPP shape of
+ // `KeyGroupedPartitioningRuntimeFilterTests`, over such a scan.
+ val catalogName = "unsupported_columnar_cat"
+ withSQLConf(
+ s"spark.sql.catalog.$catalogName" ->
classOf[UnsupportedColumnarModeCatalog].getName,
+ SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "false",
+ SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10") {
+ val modeCatalog = spark.sessionState.catalogManager.catalog(catalogName)
+ .asInstanceOf[UnsupportedColumnarModeCatalog]
+ val ns = s"$catalogName.ns"
+ try {
+ createTable(items, itemsColumns, Array(identity("id")), catalog =
modeCatalog)
+ sql(s"INSERT INTO $ns.$items VALUES " + (1 to 7).map(i =>
+ s"($i, 'x', 10.0, cast('2020-01-01' as timestamp))").mkString(", "))
+ createTable(purchases, purchasesColumns, Array(identity("item_id")),
+ catalog = modeCatalog)
+ sql(s"INSERT INTO $ns.$purchases VALUES " + (1 to 7).flatMap(i =>
+ Seq.fill(2)(s"($i, $i.0, cast('2020-01-01' as
timestamp))")).mkString(", "))
+ createTable("dim", Array(Column.create("k", LongType),
Column.create("flag", IntegerType)),
+ Array.empty, catalog = modeCatalog)
+ sql(s"INSERT INTO $ns.dim VALUES " +
+ (1 to 7).map(i => s"($i, ${if (i >= 5) 1 else 0})").mkString(", "))
+ // A dimension match, since an empty broadcast lets AQE drop the join.
+ val query = s"SELECT i.id, p.price FROM $ns.$items i JOIN " +
+ s"(SELECT /*+ BROADCAST(d) */ p.item_id, p.price FROM $ns.$purchases
p " +
+ s"JOIN $ns.dim d ON p.item_id = d.k WHERE d.flag = 1) p ON i.id =
p.item_id"
+ Seq(true, false).foreach { aqe =>
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString) {
+ val df = sql(query)
+ checkAnswer(df, (5 to 7).flatMap(i => Seq.fill(2)(Row(i.toLong,
i.toFloat))))
+ val plan = df.queryExecution.executedPlan
+ assert(collectScans(plan).forall(
+ _.scan.columnarSupportMode() ==
Scan.ColumnarSupportMode.UNSUPPORTED))
+ assert(collectShuffles(plan).isEmpty,
+ s"The join should stay storage-partitioned:\n$plan")
+ }
+ }
+ } finally {
+ modeCatalog.clearTables()
+ }
+ }
+ }
+
test("partitioned join: join with two partition keys and matching & sorted
partitions") {
val items_partitions = Array(bucket(8, "id"), days("arrive_time"))
createTable(items, itemsColumns, items_partitions)
@@ -11481,6 +11610,46 @@ case class CustomReportingScan(
override def outputOrdering(): Array[SortOrder] = ordering.toArray
}
+/**
+ * An in-memory catalog whose scans declare `UNSUPPORTED` as their columnar
support mode, so
+ * `supportsColumnar` answers without planning the scan's partitions.
+ */
+class UnsupportedColumnarModeCatalog extends InMemoryTableCatalog {
+ // scalastyle:off argcount
+ override protected def newInMemoryTable(
+ name: String,
+ columns: Array[Column],
+ partitioning: Array[Transform],
+ properties: java.util.Map[String, String],
+ constraints: Array[Constraint],
+ distribution: Distribution,
+ ordering: Array[SortOrder],
+ requiredNumPartitions: Option[Int],
+ advisoryPartitionSize: Option[Long],
+ distributionStrictlyRequired: Boolean,
+ numRowsPerSplit: Int,
+ id: String,
+ schemaAlignmentConfig: SchemaAlignmentConfig): InMemoryBaseTable = {
+ // scalastyle:on argcount
+ new InMemoryTable(name, columns, partitioning, properties, constraints,
distribution,
+ ordering, requiredNumPartitions, advisoryPartitionSize,
distributionStrictlyRequired,
+ numRowsPerSplit, id, schemaAlignmentConfig) {
+ override def newScanBuilder(options: CaseInsensitiveStringMap):
ScanBuilder =
+ new InMemoryScanBuilder(schema(), options) {
+ override protected def createScan(
+ partitions: Seq[InputPartition],
+ readSchema: StructType,
+ tableSchema: StructType,
+ options: CaseInsensitiveStringMap): BatchScanBaseClass =
+ new InMemoryBatchScan(partitions, readSchema, tableSchema,
options) {
+ override def columnarSupportMode(): Scan.ColumnarSupportMode =
+ Scan.ColumnarSupportMode.UNSUPPORTED
+ }
+ }
+ }
+ }
+}
+
/**
* A connector-defined transform whose `children()` differs from its
`arguments()`, which the
* default `Transform.children()` returns.
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
index de53f1625b37..3028159dd606 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkPlanSuite.scala
@@ -33,11 +33,12 @@ import org.apache.spark.sql.catalyst.expressions.{
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
import org.apache.spark.sql.catalyst.plans.logical.Deduplicate
import org.apache.spark.sql.catalyst.trees.LeafLike
-import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.connector.read.{Batch, InputPartition,
PartitionReaderFactory, Scan}
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
PlannedScan}
import org.apache.spark.sql.execution.exchange.ShuffleExchangeExec
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{DataType, IntegerType, StringType}
+import org.apache.spark.sql.types.{DataType, IntegerType, StringType,
StructType}
import org.apache.spark.sql.vectorized.ColumnarBatch
import org.apache.spark.util.ThreadUtils
@@ -208,14 +209,14 @@ class SparkPlanSuite extends SharedSparkSession {
// violating the contract that equal objects must have equal hash codes.
val exec1 = BatchScanExec(
output = Seq.empty,
- scan = null,
+ plannedScan = null,
runtimeFilters = Seq.empty,
table = null,
keyGroupedPartitioning = Some(Seq(Literal(1)))
)
val exec2 = BatchScanExec(
output = Seq.empty,
- scan = null,
+ plannedScan = null,
runtimeFilters = Seq.empty,
table = null,
keyGroupedPartitioning = Some(Seq(Literal(2)))
@@ -306,7 +307,7 @@ class SparkPlanSuite extends SharedSparkSession {
val data = AttributeReference("data", StringType)(ExprId(1))
val scan = BatchScanExec(
output = Seq(id, data),
- scan = null,
+ plannedScan = null,
runtimeFilters = Seq.empty,
table = null,
keyGroupedPartitioning = Some(Seq(id, data)))
@@ -321,6 +322,29 @@ class SparkPlanSuite extends SharedSparkSession {
assert(canonicalizedTypes == Seq(IntegerType, StringType),
s"Expected [IntegerType, StringType] but got $canonicalizedTypes")
}
+
+ test("SPARK-60752: copies of a BatchScanExec share its Batch and the
partitions it planned") {
+ // A `Scan` that returns a new `Batch` from each `toBatch`, which a
connector may do.
+ class NewBatchScan extends Scan {
+ override def readSchema(): StructType = new StructType()
+ override def toBatch: Batch = new Batch {
+ override def planInputPartitions(): Array[InputPartition] = Array(new
InputPartition {})
+ override def createReaderFactory(): PartitionReaderFactory =
+ throw new UnsupportedOperationException()
+ }
+ }
+ val scan = BatchScanExec(Seq.empty, new PlannedScan(new NewBatchScan),
Seq.empty, table = null)
+ def sharesPlanning(copy: SparkPlan): Boolean = copy match {
+ case c: BatchScanExec =>
+ (c.batch eq scan.batch) && (c.inputPartitions eq scan.inputPartitions)
+ case _ => false
+ }
+
+ // Copies through `makeCopy`, `clone` or `copy` share what the node
planned.
+
assert(sharesPlanning(scan.makeCopy(scan.productIterator.map(_.asInstanceOf[AnyRef]).toArray)))
+ assert(sharesPlanning(scan.clone()))
+ assert(sharesPlanning(scan.copy(runtimeFilters = Nil)))
+ }
}
case class ColumnarOp(child: SparkPlan) extends UnaryExecNode {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala
index 2ee239d9fc9f..bda520effe31 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala
@@ -27,7 +27,7 @@ import org.apache.spark.sql.catalyst.plans.logical.{Filter,
LogicalPlan, Project
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan}
import org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
-import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, FileScan}
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, FileScan,
PlannedScan}
import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
import org.apache.spark.sql.functions.broadcast
import org.apache.spark.sql.internal.SQLConf
@@ -170,7 +170,7 @@ class PruneFileSourcePartitionsSuite extends
PrunePartitionSuiteBase with Shared
override def getScanExecPartitionSize(plan: SparkPlan): Long = {
plan.collectFirst {
case p: FileSourceScanExec => p.selectedPartitions.partitionCount
- case BatchScanExec(_, scan: FileScan, _, _, _, _, _, _) =>
+ case BatchScanExec(_, PlannedScan(scan: FileScan), _, _, _, _, _, _) =>
scan.fileIndex.listFiles(scan.partitionFilters,
scan.dataFilters).length
}.get
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PrunePartitionSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PrunePartitionSuiteBase.scala
index 9a27aacedc4c..ba4e32eef1cd 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PrunePartitionSuiteBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PrunePartitionSuiteBase.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.execution.datasources
import org.apache.spark.sql.StatisticsCollectionTestBase
import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
BinaryOperator, Expression, IsNotNull, Literal}
import org.apache.spark.sql.execution.SparkPlan
-import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, FileScan}
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec, FileScan,
PlannedScan}
import org.apache.spark.sql.internal.SQLConf.ADAPTIVE_EXECUTION_ENABLED
abstract class PrunePartitionSuiteBase extends StatisticsCollectionTestBase {
@@ -96,7 +96,8 @@ abstract class PrunePartitionSuiteBase extends
StatisticsCollectionTestBase {
val collectFn: PartialFunction[SparkPlan, Seq[Expression]] =
collectPartitionFiltersFn() orElse {
- case BatchScanExec(_, scan: FileScan, _, _, _, _, _, _) =>
scan.partitionFilters
+ case BatchScanExec(_, PlannedScan(scan: FileScan), _, _, _, _, _, _) =>
+ scan.partitionFilters
}
val pushedDownPartitionFilters = plan.collectFirst(collectFn)
.map(exps => exps.filterNot(e => e.isInstanceOf[IsNotNull]))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcV2SchemaPruningSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcV2SchemaPruningSuite.scala
index eb0389f00cb2..b44909b8b391 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcV2SchemaPruningSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcV2SchemaPruningSuite.scala
@@ -21,7 +21,7 @@ import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
import org.apache.spark.sql.execution.datasources.SchemaPruningSuite
-import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
PlannedScan}
import org.apache.spark.sql.execution.datasources.v2.orc.OrcScan
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.tags.ExtendedSQLTest
@@ -42,7 +42,7 @@ class OrcV2SchemaPruningSuite extends SchemaPruningSuite with
AdaptiveSparkPlanH
override def checkScanSchemata(df: DataFrame, expectedSchemaCatalogStrings:
String*): Unit = {
val fileSourceScanSchemata =
collect(df.queryExecution.executedPlan) {
- case BatchScanExec(_, scan: OrcScan, _, _, _, _, _, _) =>
scan.readDataSchema
+ case BatchScanExec(_, PlannedScan(scan: OrcScan), _, _, _, _, _, _) =>
scan.readDataSchema
}
assert(fileSourceScanSchemata.size === expectedSchemaCatalogStrings.size,
s"Found ${fileSourceScanSchemata.size} file sources in dataframe, " +
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
index dda5c6e6baae..a00295eb43d2 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala
@@ -49,7 +49,7 @@ import org.apache.spark.sql.connector.read.{InputPartition,
PartitionReader, Par
import org.apache.spark.sql.connector.write.{BatchWrite, DataWriter,
DataWriterFactory, LogicalWriteInfo, PhysicalWriteInfo, Write, WriteBuilder}
import org.apache.spark.sql.execution.{LeafExecNode, QueryExecution,
SparkPlanInfo, SQLExecution}
import org.apache.spark.sql.execution.adaptive.DisableAdaptiveExecution
-import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.datasources.v2.{BatchScanExec,
PlannedScan}
import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
import org.apache.spark.sql.functions.count
import org.apache.spark.sql.internal.SQLConf
@@ -836,7 +836,8 @@ abstract class SQLAppStatusListenerSuite extends
SharedSparkSession with JsonTes
val statusStore = spark.sharedState.statusStore
val schema = new StructType().add("i", "int").add("j", "int")
- val physicalPlan = BatchScanExec(toAttributes(schema), new
CustomMetricScanBuilder(), Seq.empty,
+ val physicalPlan = BatchScanExec(toAttributes(schema),
+ new PlannedScan(new CustomMetricScanBuilder()), Seq.empty,
table = new TestLocalScanTable("fake"))
val dummyQueryExecution = new QueryExecution(spark, LocalRelation()) {
override lazy val sparkPlan = physicalPlan
@@ -864,8 +865,9 @@ abstract class SQLAppStatusListenerSuite extends
SharedSparkSession with JsonTes
val statusStore = spark.sharedState.statusStore
val schema = new StructType().add("i", "int").add("j", "int")
- val physicalPlan = BatchScanExec(toAttributes(schema), new
CustomDriverMetricScanBuilder(),
- Seq.empty, table = new TestLocalScanTable("fake"))
+ val physicalPlan = BatchScanExec(toAttributes(schema),
+ new PlannedScan(new CustomDriverMetricScanBuilder()), Seq.empty,
+ table = new TestLocalScanTable("fake"))
val dummyQueryExecution = new QueryExecution(spark, LocalRelation()) {
override lazy val sparkPlan = physicalPlan
override lazy val executedPlan = physicalPlan
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]