tzphh opened a new issue, #11047:
URL: https://github.com/apache/paimon/issues/11047

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   Full compaction can promote a primary-key data file to a higher LSM level by 
updating its metadata. The promoted file keeps `fileSource=APPEND`. The current 
primary-key vector index requires `fileSource=COMPACT` and `level > 0`, so 
these files remain outside ANN coverage.
   
   I observed this on a fixed-bucket `deduplicate` table with deletion vectors 
enabled. Full compaction promoted all 7,410 Parquet files from Level 0 to Level 
5, covering 1 billion physical rows. The file paths, sizes, and row counts 
stayed the same. The resulting snapshot had only Level 5 files in each bucket, 
and every file had `deleteRowCount=0`.
   
   The ordinary primary-key batch reader can already read these files without 
merging primary-key versions. The current workaround for building an IVF-FLAT 
index is to set `compaction.force-rewrite-all-files=true`, which rewrites about 
1 TB of Parquet to make the files eligible for indexing.
   
   ### Solution
   
   Support index-only backfill for files that the primary-key batch reader can 
read directly in the selected snapshot. Use the batch reader's eligibility 
rules to determine whether each visible vector can be read from a stable 
physical row position, with the applicable schema and deletion vectors.
   
   Reuse the existing file-name and row-position mapping, and commit the ANN 
index while keeping the same Parquet files and their original `fileSource`. 
Apply the same snapshot-scoped eligibility rules during index build and search. 
Files that still require primary-key merging continue through the existing 
compaction path.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to