LuciferYang commented on PR #56378: URL: https://github.com/apache/spark/pull/56378#issuecomment-4670438379
I've updated #56401. Some thoughts on the difference between the two approaches: Returning the live table here means the shared catalog loses the pin-at-load semantics from SPARK-56995. `StreamingTransactionSuite` also uses this shared catalog, and with live tables a cached reference and a fresh load are always the same instance, so `Table.equals` can never report data staleness in that suite. The schema-change tests still pass because `alterTable` replaces the map entry, but data-version staleness becomes undetectable. #56401 instead keeps the snapshots and forwards `truncateTable()` to the live table. The truncate goes through `deleteWhere` and bumps the live version, so earlier snapshots still become stale as SPARK-56995 intended. It also fixes the same bug in the non-shared catalog. BTW, I think only TRUNCATE was broken here. `DropTableExec` calls `catalog.dropTable(ident)` directly, so DROP never touched the snapshot. +1 to @AnishMahto's point that the root cause is TRUNCATE resolving without write privileges - that would need a fix in the resolution path, out of scope for both PRs. I'll leave the direction to @andreaschat-db @gengliangwang @jose-torres, and I'm fine closing #56401 if the simpler fix here is preferred. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
