This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 630c8b71dfa1 [SPARK-60863][SQL] Fix wrong results by scalar subquery
runtime filters in group-based DELETE and UPDATE
630c8b71dfa1 is described below
commit 630c8b71dfa1b036f1fa1fd60b69d8d8dd9b2101
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 06:59:17 2026 -0800
[SPARK-60863][SQL] Fix wrong results by scalar subquery runtime filters in
group-based DELETE and UPDATE
### What changes were proposed in this pull request?
This PR aims to fix a correctness issue of group-based row-level DELETE and
UPDATE. `DataSourceV2Strategy` no longer routes scalar subquery filters into
the runtime filters of a read of a group-based row-level operation, that is, a
read of a `RowLevelOperationTable` whose operation does not implement
`SupportsDelta`. As before
[SPARK-56467](https://issues.apache.org/jira/browse/SPARK-56467), scalar
subquery filters do not prune that scan at runtime. Delta-based operations keep
the routing.
### Why are the changes needed?
This is a correctness issue. A group-based DELETE leaves rows that it
should delete, and a group-based UPDATE duplicates rows, without any error. It
happens with the default configurations when the condition compares a runtime
filter column, such as an identity partition column, with a scalar subquery.
Since SPARK-56467
([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11),
`DataSourceV2Strategy` routes a deterministic filter that has a scalar subquery
and references only runtime filter columns into `BatchScanExec.runtimeFilters`.
However, a group-based operation replaces the groups that its configured scan
returns, and all reads of the operation share that scan. The read of a DELETE,
and the copy branch of an UPDATE whose condition has a subquery, filter by th
[...]
```scala
//
spark.sql.catalog.cat=org.apache.spark.sql.connector.catalog.InMemoryRowLevelOperationTableCatalog
sql("CREATE TABLE cat.ns1.t (pk INT NOT NULL, id INT, salary INT, dep
STRING) " +
"PARTITIONED BY (dep) TBLPROPERTIES ('use-catalyst-runtime-filtering' =
'true')")
sql("INSERT INTO cat.ns1.t VALUES (1, 1, 300, 'hr'), (2, 2, 150,
'software'), (3, 3, 120, 'hr')")
Seq("hr").toDF("d").createOrReplaceTempView("src")
// Issue 1: It should delete the 2 `hr` rows, but it deletes nothing.
sql("DELETE FROM cat.ns1.t WHERE dep = (SELECT max(d) FROM src)")
// Issue 2: On the same initial table, it should update the 2 `hr` rows,
but the table ends up
// with both the old and the updated `hr` rows.
sql("UPDATE cat.ns1.t SET salary = -1 WHERE dep = (SELECT max(d) FROM src)")
```
A scan is affected when it prunes with the negated filter:
- `SupportsRuntimeV2Filtering` scans since v4.2.0, e.g. a scan with
`supportsIterativePushdown()` that prunes with the `PartitionPredicate` Spark
derives from the filter.
- `SupportsRuntimeCatalystFiltering` scans, added by
[SPARK-58523](https://issues.apache.org/jira/browse/SPARK-58523)
([v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0), 2026-10-27).
Disabling
`spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled` does not
avoid the issue, because the negated filter prunes the scan by itself.
### Does this PR introduce _any_ user-facing change?
Yes. A group-based DELETE or UPDATE whose condition compares a runtime
filter column with a scalar subquery now deletes or updates the matching rows.
As in Apache Spark 4.1, scalar subquery filters do not prune its scan.
### How was this patch tested?
Pass the CIs with newly added test cases. The two test cases in
`GroupBasedRowLevelOperationCatalystRuntimeFilterSuite` fail without this PR,
because the scans get the condition or its negation as a runtime filter besides
the group filter. The test case in
`DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite` checks that a
delta-based DELETE still prunes its scan with the condition.
```
$ build/sbt 'sql/testOnly *GroupBased* *DeltaBased*
*MergeIntoDataFrameSuite *TransactionSuite *SchemaAlignmentConfigSuite
*DataSourceV2CatalystRuntimeFilterSuite
*DataSourceV2EnhancedRuntimePartitionFilterSuite
*DataSourceV2EnhancedDeleteFilterSuite
*KeyGroupedPartitioningRuntimeFilterSuite
*KeyGroupedPartitioningCatalystRuntimeFilterSuite'
$ build/sbt 'sql/testOnly *DataSourceV2SQLSuiteV2Filter -- -z SPARK-56467
-z SPARK-58207'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60079 from dongjoon-hyun/SPARK-60863.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../datasources/v2/DataSourceV2Strategy.scala | 14 +++++-
...wLevelOperationCatalystRuntimeFilterSuite.scala | 27 +++++++++++
...wLevelOperationCatalystRuntimeFilterSuite.scala | 53 ++++++++++++++++++++++
3 files changed, 92 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
index 6fddc3f31767..5ff2302bad8a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
@@ -42,7 +42,7 @@ import
org.apache.spark.sql.connector.expressions.{FieldReference, LiteralValue}
import org.apache.spark.sql.connector.expressions.filter.{AlwaysFalse, And =>
V2And, Not => V2Not, Or => V2Or, Predicate}
import org.apache.spark.sql.connector.read.LocalScan
import org.apache.spark.sql.connector.read.streaming.{ContinuousStream,
MicroBatchStream, SupportsRealTimeMode}
-import org.apache.spark.sql.connector.write.{V1Write, Write}
+import org.apache.spark.sql.connector.write.{RowLevelOperationTable,
SupportsDelta, V1Write, Write}
import org.apache.spark.sql.errors.{QueryCompilationErrors,
QueryExecutionErrors}
import org.apache.spark.sql.execution.{FilterExec, InSubqueryExec,
LeafExecNode, LocalTableScanExec, ProjectExec, RowDataSourceScanExec,
ScalarSubquery => ExecScalarSubquery, SparkPlan, SparkStrategy => Strategy}
import org.apache.spark.sql.execution.command.{CommandUtils, MetricViewHelper}
@@ -204,7 +204,17 @@ class DataSourceV2Strategy(session: SparkSession) extends
Strategy with Predicat
// pruning while the FilterExec above the scan re-evaluates them, so the
two evaluations
// may disagree and rows the source pruned away could not be recovered.
This is the
// runtime counterpart of the pushFilters guard in PushDownUtils
(SPARK-58207).
- val scalarSubqueryFilters = if (relation.runtimeFilterAttrs.nonEmpty) {
+ // Nothing is routed for a read of a group-based row-level operation
either. The operation
+ // replaces the groups its scan returns, and its reads share that scan,
so a filter of one
+ // read must not prune it: the negated condition that selects the rows
to copy over would
+ // drop the groups where every row matches. The group filter of
+ // RowLevelOperationRuntimeGroupFiltering prunes it instead, since it
keeps every group that
+ // has a matching row.
+ val groupBasedRead = relation.relation.table match {
+ case t: RowLevelOperationTable =>
!t.operation.isInstanceOf[SupportsDelta]
+ case _ => false
+ }
+ val scalarSubqueryFilters = if (relation.runtimeFilterAttrs.nonEmpty &&
!groupBasedRead) {
postScanFilters.filter { f =>
f.deterministic &&
f.containsPattern(SCALAR_SUBQUERY) &&
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
index ce104f5631f1..b7d1b6d09467 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
@@ -18,10 +18,14 @@
package org.apache.spark.sql.connector
import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.expressions.DynamicPruning
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
class DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite
extends RowLevelOperationCatalystRuntimeFilterSuiteBase {
+ import testImplicits._
+
override protected def extraTableProps: java.util.Map[String, String] = {
val props = super.extraTableProps
props.put("supports-deltas", "true")
@@ -47,4 +51,27 @@ class DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite
sql(s"SELECT * FROM $tableNameAsString"),
Row(2, 2, 150, "software") :: Row(3, 3, 120, "hr") :: Nil)
}
+
+ test("SPARK-60863: delete prunes by a scalar subquery condition on the group
key") {
+ withTempView("src") {
+ createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
+ """{ "pk": 1, "id": 1, "salary": 300, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "salary": 150, "dep": "software" }
+ |{ "pk": 3, "id": 3, "salary": 120, "dep": "hr" }
+ |""".stripMargin)
+ Seq("hr").toDF("d").createOrReplaceTempView("src")
+
+ // a delta-based delete writes only the rows it reads, so its condition
can prune the scan
+ val executedPlan = executeAndKeepPlan {
+ sql(s"DELETE FROM $tableNameAsString WHERE dep = (SELECT max(d) FROM
src)")
+ }
+ val Seq(batchScan) = collect(executedPlan) { case s: BatchScanExec => s }
+ assert(batchScan.runtimeFilters.exists(!_.isInstanceOf[DynamicPruning]),
+ s"expected the condition among the runtime filters, got
${batchScan.runtimeFilters}")
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(2, 2, 150, "software") :: Nil)
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
index 39138ceb0d39..02642a48cd44 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
@@ -26,6 +26,8 @@ import org.apache.spark.sql.connector.write.DeleteSummary
class GroupBasedRowLevelOperationCatalystRuntimeFilterSuite
extends RowLevelOperationCatalystRuntimeFilterSuiteBase {
+ import testImplicits._
+
test("delete runtime group filtering with SupportsRuntimeCatalystFiltering")
{
// the table is partitioned by dep, so hr and software are the two groups
createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
@@ -79,6 +81,57 @@ class GroupBasedRowLevelOperationCatalystRuntimeFilterSuite
checkDeleteMetrics(numDeletedRows = 1, numCopiedRows = 1)
}
+ test("SPARK-60863: delete with a scalar subquery condition on the group
key") {
+ withTempView("src") {
+ createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
+ """{ "pk": 1, "id": 1, "salary": 300, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "salary": 150, "dep": "software" }
+ |{ "pk": 3, "id": 3, "salary": 120, "dep": "hr" }
+ |""".stripMargin)
+ Seq("hr").toDF("d").createOrReplaceTempView("src")
+
+ // every hr row matches, so the negated condition that selects the rows
to copy over drops
+ // hr, and pruning the scan with it would leave hr out of the groups to
replace
+ val executedPlan = executeAndKeepPlan {
+ sql(s"DELETE FROM $tableNameAsString WHERE dep = (SELECT max(d) FROM
src)")
+ }
+ assertCatalystGroupFilter(
+ executedPlan,
+ expectedFilterPaths = Seq(Seq("dep")),
+ expectedFilter = GroupFilter(scanSchema = "dep STRING", groups =
Seq("hr")))
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(2, 2, 150, "software") :: Nil)
+ }
+ }
+
+ test("SPARK-60863: update with a scalar subquery condition on the group
key") {
+ withTempView("src") {
+ createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
+ """{ "pk": 1, "id": 1, "salary": 300, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "salary": 150, "dep": "software" }
+ |{ "pk": 3, "id": 3, "salary": 120, "dep": "hr" }
+ |""".stripMargin)
+ Seq("hr").toDF("d").createOrReplaceTempView("src")
+
+ // the subquery makes the update read the table twice, for the updated
rows and for the
+ // copied ones, and both reads share one scan, so the filter of either
read would prune it
+ // for both
+ val executedPlan = executeAndKeepPlan {
+ sql(s"UPDATE $tableNameAsString SET salary = -1 WHERE dep = (SELECT
max(d) FROM src)")
+ }
+ assertCatalystGroupFilter(
+ executedPlan,
+ expectedFilterPaths = Seq(Seq("dep")),
+ expectedFilter = GroupFilter(scanSchema = "dep STRING", groups =
Seq("hr")))
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(1, 1, -1, "hr") :: Row(2, 2, 150, "software") :: Row(3, 3, -1,
"hr") :: Nil)
+ }
+ }
+
private def checkDeleteMetrics(numDeletedRows: Long, numCopiedRows: Long):
Unit = {
val t = catalog.loadTable(ident).asInstanceOf[InMemoryTable]
val summary = t.commits.last.writeSummary.get.asInstanceOf[DeleteSummary]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]