LuciferYang opened a new issue, #10941:
URL: https://github.com/apache/paimon/issues/10941

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   master (30d91278af)
   
   ### Compute Engine
   
   Spark
   
   ### Minimal reproduce step
   
   A table with `'scan.plan-auto-tag-for-read.time-retained' = '1 h'`, read by 
two queries of the same Spark application at the same time, for example from 
two threads:
   
   ```scala
   // thread 1: a long batch read of T
   spark.sql("SELECT * FROM T").write.parquet("/tmp/out")
   
   // thread 2, while thread 1 is still reading
   spark.sql("SELECT 1").collect()
   ```
   
   ### What doesn't meet your expectations?
   
   When thread 2's query finishes, the batch read protection tag that thread 
1's scan created is deleted, although thread 1 is still reading that snapshot. 
If enough snapshots are committed and expired before thread 1 finishes, 
expiration can delete the data files thread 1 still has to read, and thread 1 
fails with `FileNotFoundException`. Protecting such reads is what the option is 
for.
   
   `BatchReadTagCleanupListener` is a `QueryExecutionListener` that keeps one 
map of pending tags, and `onSuccess` / `onFailure` of any query delete all of 
them, no matter which query planned the scan. The same happens inside one 
statement: when a command runs a nested query, the end of the nested query 
deletes the tags of the outer one.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to