dragonknightk opened a new issue, #11180:
URL: https://github.com/apache/paimon/issues/11180

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar. #5681 reports the 
same mechanism for the Spark compact procedure, which #6593 already fixed.
   
   ### Paimon version
   
   master (9b6c997)
   
   ### Compute Engine
   
   Flink
   
   ### Minimal reproduce step
   
   ```sql
   CREATE TABLE T (k INT, v INT);
   -- insert 10 rows with 10 separate INSERT statements
   ALTER TABLE T SET (
     'snapshot.expire.execution-mode' = 'async',
     'snapshot.num-retained.min' = '1',
     'snapshot.num-retained.max' = '1');
   INSERT INTO T VALUES (100, 100);  -- in batch mode, or in streaming mode 
with a bounded source
   SELECT snapshot_id FROM T$snapshots;
   ```
   
   ### What doesn't meet your expectations?
   
   Only the latest snapshot should be retained, but all 11 snapshots remain, 
both in batch mode and in streaming mode with a bounded source.
   
   With async mode, `TableCommitImpl` queues maintenance on its executor, and 
`close()` calls `shutdownNow()`, which interrupts a running expiration or drops 
one that has not started. #6593 made `commit(List)` run maintenance on the 
caller thread, which covers Spark. The end-input commit of a Flink bounded job 
uses `COMMIT_IDENTIFIER` but goes through `filterAndCommit`, so its maintenance 
is still async and is lost when the committer closes right after it. The Flink 
remove-unexisting-files action has the same pattern.
   
   ### Anything else?
   
   #5681 can likely be closed, since its Spark scenario has been fixed by #6593.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to