sarutak opened a new pull request, #56378: URL: https://github.com/apache/spark/pull/56378
### What changes were proposed in this pull request? Override loadTable in SharedTablesInMemoryRowLevelOperationTableCatalog to return the live table instance instead of a snapshot copy. SPARK-56995 (#56121) introduced a loadTable override in InMemoryRowLevelOperationTableCatalog that returns a deep copy of the table on every call. This is needed for DSv2 Transaction API cache validation semantics, but it breaks TRUNCATE TABLE and DROP TABLE IF EXISTS for tests using the shared catalog variant — the mutation is applied to a disposable copy, leaving the live table's data intact. ### Why are the changes needed? Two tests in `AutoCdcScd1FullRefreshSuite` have been failing since SPARK-56995 was merged: - "full refresh wipes target rows and the auxiliary table for the refreshed flow" - "selective full refresh wipes only the requested target's auxiliary state" https://github.com/apache/spark/actions/runs/27166004427/job/80196384953 The failures were not caught in SPARK-56995's CI because the pipelines module does not declare a dependency on sql or catalyst, so its tests were not triggered by changes under sql/catalyst/. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Confirmed all 3 tests in `AutoCdcScd1FullRefreshSuite` and all 44 tests in `MergeIntoDataFrameSuite` passed. ### Was this patch authored or co-authored using generative AI tooling? Kiro CLI / Claude -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
