steFaiz opened a new issue, #8322: URL: https://github.com/apache/paimon/issues/8322
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation In many AI workflows, users frequently need to delete data from DataEvolution tables after initial ingestion (e.g., removing low-quality samples post-feature engineering, deduplicating records, or excluding biased samples during iterative training). The existing file-level Deletion Vector (DV) in AppendTable is incompatible with DataEvolution due to dynamic row-id semantics. Rewriting files to handle random deletions is also impractical because it: Amounts to a full-table rewrite (no better than INSERT OVERWRITE) Causes an explosion of small files Invalidates external indexes since row-ids must be reassigned before compaction A tailored DV solution is therefore required for DataEvolution tables. ### Solution We propose a range-based Deletion Vector approach that maps each RowId Range to a DV instead of one DV per DataFile. Key design points include: Introduce DeletionFileKey interface to unify FileName and RowIdRange keys, reusing existing deletion file logic Maintain schema compatibility by adding a new column rather than modifying existing ones Transparently apply DVs during DataEvolutionSplitRead with no read-path performance penalty Define clear strategies for Merge Into updates, Compaction (ignore vs. materialize DV), and Vector Index adaptation ### Anything else? Please refer to [google docs](https://docs.google.com/document/d/14XHZCgtz_487eKq8k0s_hVfaVA9ETZw4rle19-qN7hY/edit?usp=sharing) for full design. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
