steFaiz opened a new pull request, #8481: URL: https://github.com/apache/paimon/pull/8481
### Purpose Parts of https://github.com/apache/paimon/issues/8322 This PR enables data-evolution compaction **WITHOUT** materializing deletion vectors. This is useful if only very small portion of data is deleted and user just want to merge small files. Note that materializing deletion vectors is very 'heavy', we need to: 1. force rewrite all columns, including blobs and vectors 2. reassign row ids 3. invalidate all global indices. So we introduce this light weight compaction. The core change: 1. After generating all commit messages, we check if any anchor file with existing DVs is compacted, if so, we rewrite the corresponding DVs to new anchor files. ### Tests See `DataEvolutionDeletionVectorTest` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
