This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b6f65ff050a5 [SPARK-60710][SQL] Expose source metadata columns to the
runtime group filter of DSv2 MERGE
b6f65ff050a5 is described below
commit b6f65ff050a57a5bceb2601f10998fa1900f2d6a
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 12:30:54 2026 -0700
[SPARK-60710][SQL] Expose source metadata columns to the runtime group
filter of DSv2 MERGE
### What changes were proposed in this pull request?
This PR aims to make the runtime group filter of DSv2 MERGE read the source
metadata columns that the `ON` condition refers to.
`RewriteMergeIntoTable.toGroupFilterCondition` converts the `ON` condition
into `EXISTS (SELECT * FROM source WHERE <ON condition>)` on top of the
original source plan. A source metadata column (e.g. `s._partition` of a DSv2
table or `s._metadata` of a file source table) is resolved from
`source.metadataOutput`, so the source plan does not output it yet when the
MERGE is rewritten. For the main join, `Analyzer.AddMetadataColumns` adds the
column to the source in the next analyzer iter [...]
```
ReplaceData RelationV2[pk#22, salary#23, dep#24] cat.ns1.test_table
: +- !Filter ((outer(pk#22) = pk#56) AND (_partition#30 = hr))
: +- SubqueryAlias s
: +- SubqueryAlias cat.ns1.source_table
: +- RelationV2[pk#56, dep#57] cat.ns1.source_table
+- MergeRows[...]
+- Join LeftOuter, ((pk#22 = pk#25) AND (_partition#30 = hr)), ...
:- ...
+- Project [pk#25, dep#26, true AS __row_from_source#34,
_partition#30]
+- SubqueryAlias s
+- SubqueryAlias cat.ns1.source_table
+- RelationV2[pk#25, dep#26, index#29, _partition#30]
cat.ns1.source_table
```
This PR adds the referenced source metadata columns to the source plan of
the subquery in the same way as `AddMetadataColumns` does, i.e.
`ExposesMetadataColumns.withMetadataColumns()` for the relation and extra
project list entries for the `Project` nodes in between. The source plan is
unchanged when the `ON` condition does not refer to source metadata columns.
```scala
+ // the condition may refer to source metadata columns that are not in
the source output
+ // AddMetadataColumns adds them to the source in the join but does not
reach this subquery
+ val sourceMetadataAttrs =
+ AttributeSet(source.metadataOutput).intersect(cond.references) --
source.outputSet
+ val filterSource = if (sourceMetadataAttrs.isEmpty) {
+ source
+ } else {
+ addMetadataAttrs(source, sourceMetadataAttrs)
+ }
+ Exists(Filter(condWithOuterRefs, filterSource), outerRefs)
```
Since the subquery's source relation now outputs the column,
`DeduplicateRelations` renews it together with the other columns, and the group
filter applies the source predicate.
```
+- BroadcastHashJoin [pk#4009], [pk#3879], LeftSemi, BuildRight, false,
false
:- Project [pk#4009, dep#4011]
: +- BatchScan cat.ns1.test_table[pk#4009, dep#4011] ...
+- BroadcastQueryStage 0
+- BroadcastExchange ...
+- *(1) Project [pk#3879]
+- *(1) Filter ((_partition#3882 = hr) AND isnotnull(pk#3879))
+- BatchScan cat.ns1.source_table[pk#3879, _partition#3882]
...
```
Two simpler alternatives were considered and not chosen.
- Skip the group filter when the `ON` condition refers to source metadata
columns. This is what disabling the group filter does today, and a group-based
MERGE reads and replaces every group of the target table.
- Drop the predicates on source metadata columns from the group filter. The
filter still selects every group with matching rows, but it can select more. In
the new test, the `software` partition would be replaced although none of its
rows changes.
### Why are the changes needed?
A DSv2 MERGE whose `ON` condition refers to a metadata column of the source
fails at analysis with an internal error when the runtime group filter is used,
i.e. `spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled` is
`true` (default) and there is no `WHEN NOT MATCHED BY SOURCE` clause.
```sql
CREATE TABLE cat.ns1.source_table (pk INT, dep STRING) PARTITIONED BY (dep);
INSERT INTO cat.ns1.source_table VALUES (1, 'hr'), (3, 'software'), (5,
'hr');
MERGE INTO cat.ns1.test_table t
USING cat.ns1.source_table s
ON t.pk = s.pk AND s._partition = 'hr'
WHEN MATCHED THEN
UPDATE SET t.salary = -1
```
```
org.apache.spark.sql.catalyst.ExtendedAnalysisException:
[MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved
attribute(s) "_partition" missing from "pk", "dep" in operator !Filter
((outer(pk#22) = pk#56) AND (_partition#30 = hr)). SQLSTATE: XX000; line 1 pos
0;
```
The same statement succeeds when the runtime group filter is disabled. A
MERGE that is rewritten in a later analyzer iteration, e.g. because its `ON`
condition needs type coercion, does not fail either, because
`AddMetadataColumns` adds the column to the MERGE source before the rewrite.
The subquery has been built on top of the original source plan since the
runtime group filter of group-based MERGE was added in Apache Spark
[3.5.0](https://github.com/apache/spark/releases/tag/v3.5.0) [...]
This is the source-side counterpart of SPARK-60668 (apache/spark#59896),
which made the runtime group filter treat the target metadata columns in the
same condition as outer references.
### Does this PR introduce _any_ user-facing change?
Yes. Such MERGE statements succeed instead of failing with the internal
error above, and the runtime group filter honors the predicates on source
metadata columns.
### How was this patch tested?
Pass the CIs with the newly added test cases. They fail without this fix.
Each test runs the statement with the DSv2 source table and with a subquery on
top of it, which covers the `Project` case. The group-based test also checks
that only the `hr` partition is replaced.
```
$ build/sbt 'sql/testOnly *GroupBasedMergeIntoTableSuite
*DeltaBasedMergeIntoTableSuite -- -z SPARK-60710'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59920 from dongjoon-hyun/SPARK-60710.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../catalyst/analysis/RewriteMergeIntoTable.scala | 31 ++++++++++++++++--
.../connector/DeltaBasedMergeIntoTableSuite.scala | 37 ++++++++++++++++++++++
.../connector/GroupBasedMergeIntoTableSuite.scala | 36 +++++++++++++++++++++
3 files changed, 102 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
index fd348e92a6dc..7d90315da730 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
@@ -22,9 +22,10 @@ import org.apache.spark.sql.catalyst.expressions.{Alias,
And, Attribute, Attribu
import org.apache.spark.sql.catalyst.expressions.Literal.{FalseLiteral,
TrueLiteral}
import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
import org.apache.spark.sql.catalyst.plans.{FullOuter, Inner, JoinType,
LeftAnti, LeftOuter, RightOuter}
-import org.apache.spark.sql.catalyst.plans.logical.{DeleteAction, Filter,
HintInfo, InsertAction, InsertOnlyMerge, Join, JoinHint, LogicalPlan,
MergeAction, MergeIntoTable, MergeRows, NO_BROADCAST_AND_REPLICATION, Project,
ReplaceData, UpdateAction, WriteDelta}
+import org.apache.spark.sql.catalyst.plans.logical.{DeleteAction,
ExposesMetadataColumns, Filter, HintInfo, InsertAction, InsertOnlyMerge, Join,
JoinHint, LogicalPlan, MergeAction, MergeIntoTable, MergeRows,
NO_BROADCAST_AND_REPLICATION, Project, ReplaceData, UpdateAction, WriteDelta}
import org.apache.spark.sql.catalyst.plans.logical.MergeRows.{Copy, Delete,
Discard, Insert, Instruction, Keep, ROW_ID, Split, Update}
import org.apache.spark.sql.catalyst.trees.TreePattern.MERGE_INTO_TABLE
+import org.apache.spark.sql.catalyst.util.MetadataColumnHelper
import org.apache.spark.sql.catalyst.util.RowDeltaUtils.{COPY_OPERATION,
INSERT_OPERATION, OPERATION_COLUMN, UPDATE_OPERATION}
import org.apache.spark.sql.connector.catalog.SupportsRowLevelOperations
import org.apache.spark.sql.connector.write.{RowLevelOperationTable,
SupportsDelta}
@@ -262,7 +263,33 @@ object RewriteMergeIntoTable extends
RewriteRowLevelCommand with PredicateHelper
val outerRefs = condWithOuterRefs.collect {
case OuterReference(e) => e
}
- Exists(Filter(condWithOuterRefs, source), outerRefs)
+ // the condition may refer to source metadata columns that are not in the
source output
+ // AddMetadataColumns adds them to the source in the join but does not
reach this subquery
+ val sourceMetadataAttrs =
+ AttributeSet(source.metadataOutput).intersect(cond.references) --
source.outputSet
+ val filterSource = if (sourceMetadataAttrs.isEmpty) {
+ source
+ } else {
+ addMetadataAttrs(source, sourceMetadataAttrs)
+ }
+ Exists(Filter(condWithOuterRefs, filterSource), outerRefs)
+ }
+
+ // adds the given metadata attributes to the output of a plan like
AddMetadataColumns does
+ private def addMetadataAttrs(plan: LogicalPlan, attrs: AttributeSet):
LogicalPlan = {
+ plan match {
+ case r: ExposesMetadataColumns if
r.metadataOutput.exists(attrs.contains) =>
+ r.withMetadataColumns()
+ case p: Project if p.metadataOutput.exists(attrs.contains) =>
+ val newAttrs = p.metadataOutput.filter(a => attrs.contains(a) &&
!p.outputSet.contains(a))
+ val newProject = p.copy(
+ projectList = p.projectList ++
newAttrs.map(_.markAsAllowAnyAccess()),
+ child = addMetadataAttrs(p.child, attrs))
+ newProject.copyTagsFrom(p)
+ newProject
+ case _ =>
+ plan.withNewChildren(plan.children.map(addMetadataAttrs(_, attrs)))
+ }
}
// build a rewrite plan for sources that support row deltas
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
index 1a477514dd10..e193669ecf33 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
@@ -75,6 +75,43 @@ class DeltaBasedMergeIntoTableSuite extends
DeltaBasedMergeIntoTableSuiteBase {
}
}
+ test("SPARK-60710: merge runtime group filtering with source metadata column
in ON condition") {
+ sql(s"CREATE TABLE $sourceNameAsString (pk INT, dep STRING) PARTITIONED BY
(dep)")
+ sql(s"INSERT INTO $sourceNameAsString VALUES (1, 'hr'), (3, 'software'),
(5, 'hr')")
+
+ Seq(sourceNameAsString, s"(SELECT pk, dep FROM
$sourceNameAsString)").foreach { source =>
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "hr" }
+ |{ "pk": 3, "salary": 300, "dep": "software" }
+ |{ "pk": 4, "salary": 400, "dep": "software" }
+ |""".stripMargin)
+
+ executeAndCheckScans(
+ s"""MERGE INTO $tableNameAsString t
+ |USING $source s
+ |ON t.pk = s.pk AND s._partition = 'hr'
+ |WHEN MATCHED THEN
+ | UPDATE SET t.salary = -1
+ |""".stripMargin,
+ primaryScanSchema = "pk INT, dep STRING, _partition STRING",
+ groupFilterScanSchema = Some("pk INT, dep STRING"))
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Seq(
+ Row(1, -1, "hr"), // update
+ Row(2, 200, "hr"), // unchanged
+ Row(3, 300, "software"), // unchanged
+ Row(4, 400, "software"))) // unchanged
+
+ checkLastWriteLog(
+ updateWriteLogEntry(id = 1, metadata = Row("hr", null), data = Row(1,
-1, "hr")))
+
+ sql(s"DROP TABLE $tableNameAsString")
+ }
+ }
+
test("merge runtime group filtering (DPP enabled)") {
withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true") {
checkMergeRuntimeGroupFiltering()
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
index 1636e88a1dcc..a79c45fdbb16 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
@@ -121,6 +121,42 @@ class GroupBasedMergeIntoTableSuite extends
MergeIntoTableSuiteBase {
}
}
+ test("SPARK-60710: merge runtime group filtering with source metadata column
in ON condition") {
+ sql(s"CREATE TABLE $sourceNameAsString (pk INT, dep STRING) PARTITIONED BY
(dep)")
+ sql(s"INSERT INTO $sourceNameAsString VALUES (1, 'hr'), (3, 'software'),
(5, 'hr')")
+
+ Seq(sourceNameAsString, s"(SELECT pk, dep FROM
$sourceNameAsString)").foreach { source =>
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "hr" }
+ |{ "pk": 3, "salary": 300, "dep": "software" }
+ |{ "pk": 4, "salary": 400, "dep": "software" }
+ |""".stripMargin)
+
+ executeAndCheckScans(
+ s"""MERGE INTO $tableNameAsString t
+ |USING $source s
+ |ON t.pk = s.pk AND s._partition = 'hr'
+ |WHEN MATCHED THEN
+ | UPDATE SET t.salary = -1
+ |""".stripMargin,
+ primaryScanSchema = "pk INT, salary INT, dep STRING, _partition
STRING, index INT",
+ groupFilterScanSchema = Some("pk INT, dep STRING"))
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Seq(
+ Row(1, -1, "hr"), // update
+ Row(2, 200, "hr"), // unchanged
+ Row(3, 300, "software"), // unchanged
+ Row(4, 400, "software"))) // unchanged
+
+ checkReplacedPartitions(Seq("hr"))
+
+ sql(s"DROP TABLE $tableNameAsString")
+ }
+ }
+
test("merge runtime group filtering (DPP enabled)") {
withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true") {
checkMergeRuntimeGroupFiltering()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]