LuciferYang opened a new issue, #10941: URL: https://github.com/apache/paimon/issues/10941
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (30d91278af) ### Compute Engine Spark ### Minimal reproduce step A table with `'scan.plan-auto-tag-for-read.time-retained' = '1 h'`, read by two queries of the same Spark application at the same time, for example from two threads: ```scala // thread 1: a long batch read of T spark.sql("SELECT * FROM T").write.parquet("/tmp/out") // thread 2, while thread 1 is still reading spark.sql("SELECT 1").collect() ``` ### What doesn't meet your expectations? When thread 2's query finishes, the batch read protection tag that thread 1's scan created is deleted, although thread 1 is still reading that snapshot. If enough snapshots are committed and expired before thread 1 finishes, expiration can delete the data files thread 1 still has to read, and thread 1 fails with `FileNotFoundException`. Protecting such reads is what the option is for. `BatchReadTagCleanupListener` is a `QueryExecutionListener` that keeps one map of pending tags, and `onSuccess` / `onFailure` of any query delete all of them, no matter which query planned the scan. The same happens inside one statement: when a command runs a nested query, the end of the nested query deletes the tags of the outer one. ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
