zhoulii opened a new issue, #11251:
URL: https://github.com/apache/paimon/issues/11251

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   DV materialization scans the index manifest during commit preparation to 
generate global index deletion messages. The core commit then replaces these 
prepared deletions using the latest snapshot on every attempt, including the 
first.
   
   The preparation step adds a redundant metadata scan and message allocations. 
Global index invalidation can be handled entirely by the existing core commit 
logic.
   
   ### Solution
   
   - Remove the preparation-time global index scan and the unused dropper.
   - Keep invalidation in the core commit, which handles concurrent index 
creation on each commit attempt.
   - Add regressions for concurrent index creation, commit retries, fully 
deleted input, and preservation of indexes in unaffected partitions.
   
   ### Anything else?
   
   This is a core optimization shared by Spark and Flink.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to