This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 06d56027a83e [SPARK-56995][SQL][DML][TESTS][FOLLOWUP] Fix
AutoCdcScd1FullRefreshSuite by returning live table from
SharedTablesInMemoryRowLevelOperationTableCatalog
06d56027a83e is described below
commit 06d56027a83e4d1eee3606889b99fe36e304d70d
Author: Kousuke Saruta <[email protected]>
AuthorDate: Thu Jun 11 06:13:11 2026 +0900
[SPARK-56995][SQL][DML][TESTS][FOLLOWUP] Fix AutoCdcScd1FullRefreshSuite by
returning live table from SharedTablesInMemoryRowLevelOperationTableCatalog
### What changes were proposed in this pull request?
Override loadTable in SharedTablesInMemoryRowLevelOperationTableCatalog to
return the live table instance instead of a snapshot copy.
SPARK-56995 (#56121) introduced a loadTable override in
InMemoryRowLevelOperationTableCatalog that returns a deep copy of the table on
every call. This is needed for DSv2 Transaction API cache validation semantics,
but it breaks TRUNCATE TABLE and DROP TABLE IF EXISTS for tests using the
shared catalog variant — the mutation is applied to a disposable copy, leaving
the live table's data intact.
### Why are the changes needed?
Two tests in `AutoCdcScd1FullRefreshSuite` have been failing since
SPARK-56995 was merged:
- "full refresh wipes target rows and the auxiliary table for the refreshed
flow"
- "selective full refresh wipes only the requested target's auxiliary state"
https://github.com/apache/spark/actions/runs/27166004427/job/80196384953
The failures were not caught in SPARK-56995's CI because the pipelines
module does not declare a dependency on sql or
catalyst, so its tests were not triggered by changes under sql/catalyst/.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Confirmed all 3 tests in `AutoCdcScd1FullRefreshSuite` and all 44 tests in
`MergeIntoDataFrameSuite` passed.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Claude
Closes #56378 from sarutak/fix-autocdc-full-refresh-suite.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
(cherry picked from commit 44984a6bfeca86e4f78d131609faee63072b05b1)
Signed-off-by: Kousuke Saruta <[email protected]>
---
.../src/test/scala/org/apache/spark/sql/connector/catalog/txns.scala | 5 +++++
1 file changed, 5 insertions(+)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/txns.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/txns.scala
index 44489a14b4fe..203aed450a5f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/txns.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/txns.scala
@@ -279,6 +279,11 @@ class SharedTablesInMemoryRowLevelOperationTableCatalog
super.initialize(name, options)
tables = SharedTablesInMemoryRowLevelOperationTableCatalog.sharedTables
}
+
+ // Return the live table instance (not a snapshot copy) so that an in-place
TRUNCATE --
+ // which resolves its target via the read-path loadTable -- mutates the
shared catalog
+ // state instead of a discarded copy. (DROP bypasses loadTable, so it is
unaffected.)
+ override def loadTable(ident: Identifier): Table = liveTable(ident)
}
object SharedTablesInMemoryRowLevelOperationTableCatalog {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
