tzphh opened a new issue, #11047: URL: https://github.com/apache/paimon/issues/11047
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Full compaction can promote a primary-key data file to a higher LSM level by updating its metadata. The promoted file keeps `fileSource=APPEND`. The current primary-key vector index requires `fileSource=COMPACT` and `level > 0`, so these files remain outside ANN coverage. I observed this on a fixed-bucket `deduplicate` table with deletion vectors enabled. Full compaction promoted all 7,410 Parquet files from Level 0 to Level 5, covering 1 billion physical rows. The file paths, sizes, and row counts stayed the same. The resulting snapshot had only Level 5 files in each bucket, and every file had `deleteRowCount=0`. The ordinary primary-key batch reader can already read these files without merging primary-key versions. The current workaround for building an IVF-FLAT index is to set `compaction.force-rewrite-all-files=true`, which rewrites about 1 TB of Parquet to make the files eligible for indexing. ### Solution Support index-only backfill for files that the primary-key batch reader can read directly in the selected snapshot. Use the batch reader's eligibility rules to determine whether each visible vector can be read from a stable physical row position, with the applicable schema and deletion vectors. Reuse the existing file-name and row-position mapping, and commit the ANN index while keeping the same Parquet files and their original `fileSource`. Apply the same snapshot-scoped eligibility rules during index build and search. Files that still require primary-key merging continue through the existing compaction path. ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
