LuciferYang opened a new pull request, #56401: URL: https://github.com/apache/spark/pull/56401
### What changes were proposed in this pull request? This PR updates `InMemoryRowLevelOperationTable.copy()` so snapshot copies created by `loadTable` can forward `truncateTable()` to the catalog's live table. ### Why are the changes needed? SPARK-56995 changed the row-level test catalog to return snapshot copies from `loadTable` for cache-staleness testing. That is correct for read planning, but DDL `TRUNCATE TABLE` must mutate the live catalog table. Otherwise full refresh tests can truncate only the resolved snapshot and leave stale rows in the backing table. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Before this patch, the suite failed on upstream master: ```bash build/sbt 'pipelines/testOnly org.apache.spark.sql.pipelines.graph.AutoCdcScd1FullRefreshSuite' ``` ```text AutoCdcScd1FullRefreshSuite: - full refresh wipes target rows and the auxiliary table for the refreshed flow *** FAILED *** sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: Results do not match for query: ... !== Correct Answer - 1 == == Spark Answer - 2 == !struct<> struct<id:int,name:string,version:bigint,__spark_autocdc_metadata:struct<deleteSequence:bigint,upsertSequence:bigint>> ![2,bob,1,[null,1]] [1,alice,5,[null,5]] ! [2,bob,1,[null,1]] - selective full refresh wipes only the requested target's auxiliary state *** FAILED *** sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: Results do not match for query: ... !== Correct Answer - 1 == == Spark Answer - 1 == !struct<> struct<id:int,version:bigint,__spark_autocdc_metadata:struct<deleteSequence:bigint,upsertSequence:bigint>> ![1,5,[null,5]] [1,10,[null,10]] ``` After this patch: ```bash build/sbt 'pipelines/testOnly org.apache.spark.sql.pipelines.graph.AutoCdcScd1FullRefreshSuite' ``` ```text AutoCdcScd1FullRefreshSuite: - full refresh wipes target rows and the auxiliary table for the refreshed flow - after a full refresh, an event with a sequence below the previous run's watermark now lands - selective full refresh wipes only the requested target's auxiliary state Tests: succeeded 3, failed 0 ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
