This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 630c8b71dfa1 [SPARK-60863][SQL] Fix wrong results by scalar subquery 
runtime filters in group-based DELETE and UPDATE
630c8b71dfa1 is described below

commit 630c8b71dfa1b036f1fa1fd60b69d8d8dd9b2101
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 06:59:17 2026 -0800

    [SPARK-60863][SQL] Fix wrong results by scalar subquery runtime filters in 
group-based DELETE and UPDATE
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix a correctness issue of group-based row-level DELETE and 
UPDATE. `DataSourceV2Strategy` no longer routes scalar subquery filters into 
the runtime filters of a read of a group-based row-level operation, that is, a 
read of a `RowLevelOperationTable` whose operation does not implement 
`SupportsDelta`. As before 
[SPARK-56467](https://issues.apache.org/jira/browse/SPARK-56467), scalar 
subquery filters do not prune that scan at runtime. Delta-based operations keep 
the routing.
    
    ### Why are the changes needed?
    
    This is a correctness issue. A group-based DELETE leaves rows that it 
should delete, and a group-based UPDATE duplicates rows, without any error. It 
happens with the default configurations when the condition compares a runtime 
filter column, such as an identity partition column, with a scalar subquery.
    
    Since SPARK-56467 
([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11), 
`DataSourceV2Strategy` routes a deterministic filter that has a scalar subquery 
and references only runtime filter columns into `BatchScanExec.runtimeFilters`. 
However, a group-based operation replaces the groups that its configured scan 
returns, and all reads of the operation share that scan. The read of a DELETE, 
and the copy branch of an UPDATE whose condition has a subquery, filter by th 
[...]
    
    ```scala
    // 
spark.sql.catalog.cat=org.apache.spark.sql.connector.catalog.InMemoryRowLevelOperationTableCatalog
    sql("CREATE TABLE cat.ns1.t (pk INT NOT NULL, id INT, salary INT, dep 
STRING) " +
      "PARTITIONED BY (dep) TBLPROPERTIES ('use-catalyst-runtime-filtering' = 
'true')")
    sql("INSERT INTO cat.ns1.t VALUES (1, 1, 300, 'hr'), (2, 2, 150, 
'software'), (3, 3, 120, 'hr')")
    Seq("hr").toDF("d").createOrReplaceTempView("src")
    
    // Issue 1: It should delete the 2 `hr` rows, but it deletes nothing.
    sql("DELETE FROM cat.ns1.t WHERE dep = (SELECT max(d) FROM src)")
    
    // Issue 2: On the same initial table, it should update the 2 `hr` rows, 
but the table ends up
    // with both the old and the updated `hr` rows.
    sql("UPDATE cat.ns1.t SET salary = -1 WHERE dep = (SELECT max(d) FROM src)")
    ```
    
    A scan is affected when it prunes with the negated filter:
    - `SupportsRuntimeV2Filtering` scans since v4.2.0, e.g. a scan with 
`supportsIterativePushdown()` that prunes with the `PartitionPredicate` Spark 
derives from the filter.
    - `SupportsRuntimeCatalystFiltering` scans, added by 
[SPARK-58523](https://issues.apache.org/jira/browse/SPARK-58523) 
([v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0), 2026-10-27).
    
    Disabling 
`spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled` does not 
avoid the issue, because the negated filter prunes the scan by itself.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A group-based DELETE or UPDATE whose condition compares a runtime 
filter column with a scalar subquery now deletes or updates the matching rows. 
As in Apache Spark 4.1, scalar subquery filters do not prune its scan.
    
    ### How was this patch tested?
    
    Pass the CIs with newly added test cases. The two test cases in 
`GroupBasedRowLevelOperationCatalystRuntimeFilterSuite` fail without this PR, 
because the scans get the condition or its negation as a runtime filter besides 
the group filter. The test case in 
`DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite` checks that a 
delta-based DELETE still prunes its scan with the condition.
    
    ```
    $ build/sbt 'sql/testOnly *GroupBased* *DeltaBased* 
*MergeIntoDataFrameSuite *TransactionSuite *SchemaAlignmentConfigSuite 
*DataSourceV2CatalystRuntimeFilterSuite 
*DataSourceV2EnhancedRuntimePartitionFilterSuite 
*DataSourceV2EnhancedDeleteFilterSuite 
*KeyGroupedPartitioningRuntimeFilterSuite 
*KeyGroupedPartitioningCatalystRuntimeFilterSuite'
    $ build/sbt 'sql/testOnly *DataSourceV2SQLSuiteV2Filter -- -z SPARK-56467 
-z SPARK-58207'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60079 from dongjoon-hyun/SPARK-60863.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../datasources/v2/DataSourceV2Strategy.scala      | 14 +++++-
 ...wLevelOperationCatalystRuntimeFilterSuite.scala | 27 +++++++++++
 ...wLevelOperationCatalystRuntimeFilterSuite.scala | 53 ++++++++++++++++++++++
 3 files changed, 92 insertions(+), 2 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
index 6fddc3f31767..5ff2302bad8a 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
@@ -42,7 +42,7 @@ import 
org.apache.spark.sql.connector.expressions.{FieldReference, LiteralValue}
 import org.apache.spark.sql.connector.expressions.filter.{AlwaysFalse, And => 
V2And, Not => V2Not, Or => V2Or, Predicate}
 import org.apache.spark.sql.connector.read.LocalScan
 import org.apache.spark.sql.connector.read.streaming.{ContinuousStream, 
MicroBatchStream, SupportsRealTimeMode}
-import org.apache.spark.sql.connector.write.{V1Write, Write}
+import org.apache.spark.sql.connector.write.{RowLevelOperationTable, 
SupportsDelta, V1Write, Write}
 import org.apache.spark.sql.errors.{QueryCompilationErrors, 
QueryExecutionErrors}
 import org.apache.spark.sql.execution.{FilterExec, InSubqueryExec, 
LeafExecNode, LocalTableScanExec, ProjectExec, RowDataSourceScanExec, 
ScalarSubquery => ExecScalarSubquery, SparkPlan, SparkStrategy => Strategy}
 import org.apache.spark.sql.execution.command.{CommandUtils, MetricViewHelper}
@@ -204,7 +204,17 @@ class DataSourceV2Strategy(session: SparkSession) extends 
Strategy with Predicat
       // pruning while the FilterExec above the scan re-evaluates them, so the 
two evaluations
       // may disagree and rows the source pruned away could not be recovered. 
This is the
       // runtime counterpart of the pushFilters guard in PushDownUtils 
(SPARK-58207).
-      val scalarSubqueryFilters = if (relation.runtimeFilterAttrs.nonEmpty) {
+      // Nothing is routed for a read of a group-based row-level operation 
either. The operation
+      // replaces the groups its scan returns, and its reads share that scan, 
so a filter of one
+      // read must not prune it: the negated condition that selects the rows 
to copy over would
+      // drop the groups where every row matches. The group filter of
+      // RowLevelOperationRuntimeGroupFiltering prunes it instead, since it 
keeps every group that
+      // has a matching row.
+      val groupBasedRead = relation.relation.table match {
+        case t: RowLevelOperationTable => 
!t.operation.isInstanceOf[SupportsDelta]
+        case _ => false
+      }
+      val scalarSubqueryFilters = if (relation.runtimeFilterAttrs.nonEmpty && 
!groupBasedRead) {
         postScanFilters.filter { f =>
           f.deterministic &&
             f.containsPattern(SCALAR_SUBQUERY) &&
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
index ce104f5631f1..b7d1b6d09467 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
@@ -18,10 +18,14 @@
 package org.apache.spark.sql.connector
 
 import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.expressions.DynamicPruning
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
 
 class DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite
   extends RowLevelOperationCatalystRuntimeFilterSuiteBase {
 
+  import testImplicits._
+
   override protected def extraTableProps: java.util.Map[String, String] = {
     val props = super.extraTableProps
     props.put("supports-deltas", "true")
@@ -47,4 +51,27 @@ class DeltaBasedRowLevelOperationCatalystRuntimeFilterSuite
       sql(s"SELECT * FROM $tableNameAsString"),
       Row(2, 2, 150, "software") :: Row(3, 3, 120, "hr") :: Nil)
   }
+
+  test("SPARK-60863: delete prunes by a scalar subquery condition on the group 
key") {
+    withTempView("src") {
+      createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
+        """{ "pk": 1, "id": 1, "salary": 300, "dep": "hr" }
+          |{ "pk": 2, "id": 2, "salary": 150, "dep": "software" }
+          |{ "pk": 3, "id": 3, "salary": 120, "dep": "hr" }
+          |""".stripMargin)
+      Seq("hr").toDF("d").createOrReplaceTempView("src")
+
+      // a delta-based delete writes only the rows it reads, so its condition 
can prune the scan
+      val executedPlan = executeAndKeepPlan {
+        sql(s"DELETE FROM $tableNameAsString WHERE dep = (SELECT max(d) FROM 
src)")
+      }
+      val Seq(batchScan) = collect(executedPlan) { case s: BatchScanExec => s }
+      assert(batchScan.runtimeFilters.exists(!_.isInstanceOf[DynamicPruning]),
+        s"expected the condition among the runtime filters, got 
${batchScan.runtimeFilters}")
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Row(2, 2, 150, "software") :: Nil)
+    }
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
index 39138ceb0d39..02642a48cd44 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedRowLevelOperationCatalystRuntimeFilterSuite.scala
@@ -26,6 +26,8 @@ import org.apache.spark.sql.connector.write.DeleteSummary
 class GroupBasedRowLevelOperationCatalystRuntimeFilterSuite
   extends RowLevelOperationCatalystRuntimeFilterSuiteBase {
 
+  import testImplicits._
+
   test("delete runtime group filtering with SupportsRuntimeCatalystFiltering") 
{
     // the table is partitioned by dep, so hr and software are the two groups
     createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
@@ -79,6 +81,57 @@ class GroupBasedRowLevelOperationCatalystRuntimeFilterSuite
     checkDeleteMetrics(numDeletedRows = 1, numCopiedRows = 1)
   }
 
+  test("SPARK-60863: delete with a scalar subquery condition on the group 
key") {
+    withTempView("src") {
+      createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
+        """{ "pk": 1, "id": 1, "salary": 300, "dep": "hr" }
+          |{ "pk": 2, "id": 2, "salary": 150, "dep": "software" }
+          |{ "pk": 3, "id": 3, "salary": 120, "dep": "hr" }
+          |""".stripMargin)
+      Seq("hr").toDF("d").createOrReplaceTempView("src")
+
+      // every hr row matches, so the negated condition that selects the rows 
to copy over drops
+      // hr, and pruning the scan with it would leave hr out of the groups to 
replace
+      val executedPlan = executeAndKeepPlan {
+        sql(s"DELETE FROM $tableNameAsString WHERE dep = (SELECT max(d) FROM 
src)")
+      }
+      assertCatalystGroupFilter(
+        executedPlan,
+        expectedFilterPaths = Seq(Seq("dep")),
+        expectedFilter = GroupFilter(scanSchema = "dep STRING", groups = 
Seq("hr")))
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Row(2, 2, 150, "software") :: Nil)
+    }
+  }
+
+  test("SPARK-60863: update with a scalar subquery condition on the group 
key") {
+    withTempView("src") {
+      createAndInitTable("pk INT NOT NULL, id INT, salary INT, dep STRING",
+        """{ "pk": 1, "id": 1, "salary": 300, "dep": "hr" }
+          |{ "pk": 2, "id": 2, "salary": 150, "dep": "software" }
+          |{ "pk": 3, "id": 3, "salary": 120, "dep": "hr" }
+          |""".stripMargin)
+      Seq("hr").toDF("d").createOrReplaceTempView("src")
+
+      // the subquery makes the update read the table twice, for the updated 
rows and for the
+      // copied ones, and both reads share one scan, so the filter of either 
read would prune it
+      // for both
+      val executedPlan = executeAndKeepPlan {
+        sql(s"UPDATE $tableNameAsString SET salary = -1 WHERE dep = (SELECT 
max(d) FROM src)")
+      }
+      assertCatalystGroupFilter(
+        executedPlan,
+        expectedFilterPaths = Seq(Seq("dep")),
+        expectedFilter = GroupFilter(scanSchema = "dep STRING", groups = 
Seq("hr")))
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Row(1, 1, -1, "hr") :: Row(2, 2, 150, "software") :: Row(3, 3, -1, 
"hr") :: Nil)
+    }
+  }
+
   private def checkDeleteMetrics(numDeletedRows: Long, numCopiedRows: Long): 
Unit = {
     val t = catalog.loadTable(ident).asInstanceOf[InMemoryTable]
     val summary = t.commits.last.writeSummary.get.asInstanceOf[DeleteSummary]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to