This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new aee63eb81b98 [SPARK-60710][SQL] Expose source metadata columns to the 
runtime group filter of DSv2 MERGE
aee63eb81b98 is described below

commit aee63eb81b98ebf3c1492fd221b23be7b58aad7d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 12:30:54 2026 -0700

    [SPARK-60710][SQL] Expose source metadata columns to the runtime group 
filter of DSv2 MERGE
    
    ### What changes were proposed in this pull request?
    
    This PR aims to make the runtime group filter of DSv2 MERGE read the source 
metadata columns that the `ON` condition refers to.
    
    `RewriteMergeIntoTable.toGroupFilterCondition` converts the `ON` condition 
into `EXISTS (SELECT * FROM source WHERE <ON condition>)` on top of the 
original source plan. A source metadata column (e.g. `s._partition` of a DSv2 
table or `s._metadata` of a file source table) is resolved from 
`source.metadataOutput`, so the source plan does not output it yet when the 
MERGE is rewritten. For the main join, `Analyzer.AddMetadataColumns` adds the 
column to the source in the next analyzer iter [...]
    
    ```
    ReplaceData RelationV2[pk#22, salary#23, dep#24] cat.ns1.test_table
    :  +- !Filter ((outer(pk#22) = pk#56) AND (_partition#30 = hr))
    :     +- SubqueryAlias s
    :        +- SubqueryAlias cat.ns1.source_table
    :           +- RelationV2[pk#56, dep#57] cat.ns1.source_table
    +- MergeRows[...]
       +- Join LeftOuter, ((pk#22 = pk#25) AND (_partition#30 = hr)), ...
          :- ...
          +- Project [pk#25, dep#26, true AS __row_from_source#34, 
_partition#30]
             +- SubqueryAlias s
                +- SubqueryAlias cat.ns1.source_table
                   +- RelationV2[pk#25, dep#26, index#29, _partition#30] 
cat.ns1.source_table
    ```
    
    This PR adds the referenced source metadata columns to the source plan of 
the subquery in the same way as `AddMetadataColumns` does, i.e. 
`ExposesMetadataColumns.withMetadataColumns()` for the relation and extra 
project list entries for the `Project` nodes in between. The source plan is 
unchanged when the `ON` condition does not refer to source metadata columns.
    
    ```scala
    +    // the condition may refer to source metadata columns that are not in 
the source output
    +    // AddMetadataColumns adds them to the source in the join but does not 
reach this subquery
    +    val sourceMetadataAttrs =
    +      AttributeSet(source.metadataOutput).intersect(cond.references) -- 
source.outputSet
    +    val filterSource = if (sourceMetadataAttrs.isEmpty) {
    +      source
    +    } else {
    +      addMetadataAttrs(source, sourceMetadataAttrs)
    +    }
    +    Exists(Filter(condWithOuterRefs, filterSource), outerRefs)
    ```
    
    Since the subquery's source relation now outputs the column, 
`DeduplicateRelations` renews it together with the other columns, and the group 
filter applies the source predicate.
    
    ```
    +- BroadcastHashJoin [pk#4009], [pk#3879], LeftSemi, BuildRight, false, 
false
       :- Project [pk#4009, dep#4011]
       :  +- BatchScan cat.ns1.test_table[pk#4009, dep#4011] ...
       +- BroadcastQueryStage 0
          +- BroadcastExchange ...
             +- *(1) Project [pk#3879]
                +- *(1) Filter ((_partition#3882 = hr) AND isnotnull(pk#3879))
                   +- BatchScan cat.ns1.source_table[pk#3879, _partition#3882] 
...
    ```
    
    Two simpler alternatives were considered and not chosen.
    - Skip the group filter when the `ON` condition refers to source metadata 
columns. This is what disabling the group filter does today, and a group-based 
MERGE reads and replaces every group of the target table.
    - Drop the predicates on source metadata columns from the group filter. The 
filter still selects every group with matching rows, but it can select more. In 
the new test, the `software` partition would be replaced although none of its 
rows changes.
    
    ### Why are the changes needed?
    
    A DSv2 MERGE whose `ON` condition refers to a metadata column of the source 
fails at analysis with an internal error when the runtime group filter is used, 
i.e. `spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled` is 
`true` (default) and there is no `WHEN NOT MATCHED BY SOURCE` clause.
    
    ```sql
    CREATE TABLE cat.ns1.source_table (pk INT, dep STRING) PARTITIONED BY (dep);
    INSERT INTO cat.ns1.source_table VALUES (1, 'hr'), (3, 'software'), (5, 
'hr');
    
    MERGE INTO cat.ns1.test_table t
    USING cat.ns1.source_table s
    ON t.pk = s.pk AND s._partition = 'hr'
    WHEN MATCHED THEN
     UPDATE SET t.salary = -1
    ```
    
    ```
    org.apache.spark.sql.catalyst.ExtendedAnalysisException: 
[MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved 
attribute(s) "_partition" missing from "pk", "dep" in operator !Filter 
((outer(pk#22) = pk#56) AND (_partition#30 = hr)).  SQLSTATE: XX000; line 1 pos 
0;
    ```
    
    The same statement succeeds when the runtime group filter is disabled. A 
MERGE that is rewritten in a later analyzer iteration, e.g. because its `ON` 
condition needs type coercion, does not fail either, because 
`AddMetadataColumns` adds the column to the MERGE source before the rewrite. 
The subquery has been built on top of the original source plan since the 
runtime group filter of group-based MERGE was added in Apache Spark 
[3.5.0](https://github.com/apache/spark/releases/tag/v3.5.0) [...]
    
    This is the source-side counterpart of SPARK-60668 (apache/spark#59896), 
which made the runtime group filter treat the target metadata columns in the 
same condition as outer references.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Such MERGE statements succeed instead of failing with the internal 
error above, and the runtime group filter honors the predicates on source 
metadata columns.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases. They fail without this fix. 
Each test runs the statement with the DSv2 source table and with a subquery on 
top of it, which covers the `Project` case. The group-based test also checks 
that only the `hr` partition is replaced.
    
    ```
    $ build/sbt 'sql/testOnly *GroupBasedMergeIntoTableSuite 
*DeltaBasedMergeIntoTableSuite -- -z SPARK-60710'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59920 from dongjoon-hyun/SPARK-60710.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit b6f65ff050a57a5bceb2601f10998fa1900f2d6a)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../catalyst/analysis/RewriteMergeIntoTable.scala  | 31 ++++++++++++++++--
 .../connector/DeltaBasedMergeIntoTableSuite.scala  | 37 ++++++++++++++++++++++
 .../connector/GroupBasedMergeIntoTableSuite.scala  | 36 +++++++++++++++++++++
 3 files changed, 102 insertions(+), 2 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
index fd348e92a6dc..7d90315da730 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
@@ -22,9 +22,10 @@ import org.apache.spark.sql.catalyst.expressions.{Alias, 
And, Attribute, Attribu
 import org.apache.spark.sql.catalyst.expressions.Literal.{FalseLiteral, 
TrueLiteral}
 import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
 import org.apache.spark.sql.catalyst.plans.{FullOuter, Inner, JoinType, 
LeftAnti, LeftOuter, RightOuter}
-import org.apache.spark.sql.catalyst.plans.logical.{DeleteAction, Filter, 
HintInfo, InsertAction, InsertOnlyMerge, Join, JoinHint, LogicalPlan, 
MergeAction, MergeIntoTable, MergeRows, NO_BROADCAST_AND_REPLICATION, Project, 
ReplaceData, UpdateAction, WriteDelta}
+import org.apache.spark.sql.catalyst.plans.logical.{DeleteAction, 
ExposesMetadataColumns, Filter, HintInfo, InsertAction, InsertOnlyMerge, Join, 
JoinHint, LogicalPlan, MergeAction, MergeIntoTable, MergeRows, 
NO_BROADCAST_AND_REPLICATION, Project, ReplaceData, UpdateAction, WriteDelta}
 import org.apache.spark.sql.catalyst.plans.logical.MergeRows.{Copy, Delete, 
Discard, Insert, Instruction, Keep, ROW_ID, Split, Update}
 import org.apache.spark.sql.catalyst.trees.TreePattern.MERGE_INTO_TABLE
+import org.apache.spark.sql.catalyst.util.MetadataColumnHelper
 import org.apache.spark.sql.catalyst.util.RowDeltaUtils.{COPY_OPERATION, 
INSERT_OPERATION, OPERATION_COLUMN, UPDATE_OPERATION}
 import org.apache.spark.sql.connector.catalog.SupportsRowLevelOperations
 import org.apache.spark.sql.connector.write.{RowLevelOperationTable, 
SupportsDelta}
@@ -262,7 +263,33 @@ object RewriteMergeIntoTable extends 
RewriteRowLevelCommand with PredicateHelper
     val outerRefs = condWithOuterRefs.collect {
       case OuterReference(e) => e
     }
-    Exists(Filter(condWithOuterRefs, source), outerRefs)
+    // the condition may refer to source metadata columns that are not in the 
source output
+    // AddMetadataColumns adds them to the source in the join but does not 
reach this subquery
+    val sourceMetadataAttrs =
+      AttributeSet(source.metadataOutput).intersect(cond.references) -- 
source.outputSet
+    val filterSource = if (sourceMetadataAttrs.isEmpty) {
+      source
+    } else {
+      addMetadataAttrs(source, sourceMetadataAttrs)
+    }
+    Exists(Filter(condWithOuterRefs, filterSource), outerRefs)
+  }
+
+  // adds the given metadata attributes to the output of a plan like 
AddMetadataColumns does
+  private def addMetadataAttrs(plan: LogicalPlan, attrs: AttributeSet): 
LogicalPlan = {
+    plan match {
+      case r: ExposesMetadataColumns if 
r.metadataOutput.exists(attrs.contains) =>
+        r.withMetadataColumns()
+      case p: Project if p.metadataOutput.exists(attrs.contains) =>
+        val newAttrs = p.metadataOutput.filter(a => attrs.contains(a) && 
!p.outputSet.contains(a))
+        val newProject = p.copy(
+          projectList = p.projectList ++ 
newAttrs.map(_.markAsAllowAnyAccess()),
+          child = addMetadataAttrs(p.child, attrs))
+        newProject.copyTagsFrom(p)
+        newProject
+      case _ =>
+        plan.withNewChildren(plan.children.map(addMetadataAttrs(_, attrs)))
+    }
   }
 
   // build a rewrite plan for sources that support row deltas
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
index 1a477514dd10..e193669ecf33 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
@@ -75,6 +75,43 @@ class DeltaBasedMergeIntoTableSuite extends 
DeltaBasedMergeIntoTableSuiteBase {
     }
   }
 
+  test("SPARK-60710: merge runtime group filtering with source metadata column 
in ON condition") {
+    sql(s"CREATE TABLE $sourceNameAsString (pk INT, dep STRING) PARTITIONED BY 
(dep)")
+    sql(s"INSERT INTO $sourceNameAsString VALUES (1, 'hr'), (3, 'software'), 
(5, 'hr')")
+
+    Seq(sourceNameAsString, s"(SELECT pk, dep FROM 
$sourceNameAsString)").foreach { source =>
+      createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+        """{ "pk": 1, "salary": 100, "dep": "hr" }
+          |{ "pk": 2, "salary": 200, "dep": "hr" }
+          |{ "pk": 3, "salary": 300, "dep": "software" }
+          |{ "pk": 4, "salary": 400, "dep": "software" }
+          |""".stripMargin)
+
+      executeAndCheckScans(
+        s"""MERGE INTO $tableNameAsString t
+           |USING $source s
+           |ON t.pk = s.pk AND s._partition = 'hr'
+           |WHEN MATCHED THEN
+           | UPDATE SET t.salary = -1
+           |""".stripMargin,
+        primaryScanSchema = "pk INT, dep STRING, _partition STRING",
+        groupFilterScanSchema = Some("pk INT, dep STRING"))
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Seq(
+          Row(1, -1, "hr"), // update
+          Row(2, 200, "hr"), // unchanged
+          Row(3, 300, "software"), // unchanged
+          Row(4, 400, "software"))) // unchanged
+
+      checkLastWriteLog(
+        updateWriteLogEntry(id = 1, metadata = Row("hr", null), data = Row(1, 
-1, "hr")))
+
+      sql(s"DROP TABLE $tableNameAsString")
+    }
+  }
+
   test("merge runtime group filtering (DPP enabled)") {
     withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true") {
       checkMergeRuntimeGroupFiltering()
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
index 1636e88a1dcc..a79c45fdbb16 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
@@ -121,6 +121,42 @@ class GroupBasedMergeIntoTableSuite extends 
MergeIntoTableSuiteBase {
     }
   }
 
+  test("SPARK-60710: merge runtime group filtering with source metadata column 
in ON condition") {
+    sql(s"CREATE TABLE $sourceNameAsString (pk INT, dep STRING) PARTITIONED BY 
(dep)")
+    sql(s"INSERT INTO $sourceNameAsString VALUES (1, 'hr'), (3, 'software'), 
(5, 'hr')")
+
+    Seq(sourceNameAsString, s"(SELECT pk, dep FROM 
$sourceNameAsString)").foreach { source =>
+      createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+        """{ "pk": 1, "salary": 100, "dep": "hr" }
+          |{ "pk": 2, "salary": 200, "dep": "hr" }
+          |{ "pk": 3, "salary": 300, "dep": "software" }
+          |{ "pk": 4, "salary": 400, "dep": "software" }
+          |""".stripMargin)
+
+      executeAndCheckScans(
+        s"""MERGE INTO $tableNameAsString t
+           |USING $source s
+           |ON t.pk = s.pk AND s._partition = 'hr'
+           |WHEN MATCHED THEN
+           | UPDATE SET t.salary = -1
+           |""".stripMargin,
+        primaryScanSchema = "pk INT, salary INT, dep STRING, _partition 
STRING, index INT",
+        groupFilterScanSchema = Some("pk INT, dep STRING"))
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Seq(
+          Row(1, -1, "hr"), // update
+          Row(2, 200, "hr"), // unchanged
+          Row(3, 300, "software"), // unchanged
+          Row(4, 400, "software"))) // unchanged
+
+      checkReplacedPartitions(Seq("hr"))
+
+      sql(s"DROP TABLE $tableNameAsString")
+    }
+  }
+
   test("merge runtime group filtering (DPP enabled)") {
     withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true") {
       checkMergeRuntimeGroupFiltering()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to