johnnyzhon opened a new issue, #59627: URL: https://github.com/apache/spark/issues/59627
We observed stale cached query results after successful replacement of a UC managed Delta table on Spark 4.0.1 with Delta 4.2.0 and OSS Unity Catalog 0.6.0. Both SQL REPLACE TABLE AS SELECT and DataFrameWriter.saveAsTable(mode="overwrite") are affected. The writes and reads occur in the same Spark session; this is not an external writer requiring explicit refresh. ### Reproduction workflow Precondition: a configured UC catalog supporting managed Delta tables, Java 17, and the matching Spark-line Delta/UC artifacts. Use independent tables for the two variants. 1. Create a UC managed Delta table containing `(1, 'original')`, with `delta.feature.catalogManaged=supported`, deletion vectors disabled and CDF disabled. 2. Cache the qualified table name and materialize its rows. For the RTAS variant, also cache and materialize a dependent temporary view. 3. Replace the table with `(2, 'replacement'), (3, 'replacement')` through RTAS, or overwrite the independently seeded table with `(3, 'overwritten')` through saveAsTable. 4. Read the qualified table name through a newly created query, without REFRESH TABLE, CLEAR CACHE or manual uncache. Expected: the new query returns the replacement rows; the dependent cached view also reflects the replacement. Actual on Spark 4.0.1: Delta history advances to version 1, but the fresh query still returns `(1, 'original')`. The dependent view in the RTAS variant is stale too. No persisted-data loss has been observed. ### Verification | Spark | Delta | Execution | RTAS | saveAsTable overwrite | Append control | | --- | --- | --- | --- | --- | --- | | 4.0.1 | 4.2.0 | local[4] | Stale | Stale | Fresh | | 4.0.1 | 4.2.0 | Standalone, separate driver/executor JVMs | Stale | Stale | Fresh | | 4.1.1 | 4.2.0 | local[4] | Fresh | Fresh | Fresh | All controls assert empty `spark.plugins`. The RAPIDS plugin is not activated, although its jar remains on the classpath for a plan-capture listener, and an NVIDIA integration-tests jar supplies the UC filesystem fixture. We have not yet reproduced this in a distribution containing no NVIDIA jars. The Spark-line comparison uses the corresponding Delta and UC artifacts, not byte-identical dependencies. ### Diagnosis and question The cached analyzed plan contains a V1 LogicalRelation. Spark 4.0.1's DataSourceV2Strategy constructs a synthetic DataSourceV2Relation for invalidation; it does not match the cached plan (`sameResult == false`, cache lookup empty). Both writes execute AtomicReplaceTableAsSelectExec. Repeating relation-based uncache after recording the stale read leaves the cache intact; qualified-name uncache clears it and the next read returns the committed replacement rows. PR #52599, whose stated purpose is TimeTravelSpec support, also changes this strategy to qualified-name invalidation. We reference that code change as a possible explanation for the passing Spark 4.1.1 control, not as confirmation that this specific defect was reported or fixed there. We have not tested an isolated backport. Please confirm ownership of the V1/V2 invalidation mismatch and whether a focused correction for Spark 4.0.x is appropriate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
