SteNicholas opened a new issue, #1381:
URL: https://github.com/apache/paimon-rust/issues/1381

   ### Search before asking
   
   - [x] I searched existing issues and pull requests and found no issue 
tracking dedicated Data Evolution Vector file compaction.
   
   ### Motivation
   
   Multimodal embedding updates create small dedicated Vector files and 
overlapping field versions. Rust's normal Data Evolution compaction leaves them 
untouched, so applications cannot consolidate this storage through the Rust 
maintenance pipeline.
   
   The existing normal-file implementation (#1249) [explicitly excludes 
dedicated BLOB/Vector 
compaction](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/docs/src/compaction.md#L647-L699).
 This issue is the suggested follow-up to #1380; reuse the dedicated-task and 
commit infrastructure established for BLOB first.
   
   Scope: Data Evolution append tables with row tracking, no primary key, 
`bucket=-1`, and `vector.file.format` configured. This concerns row-ID-aligned 
Vector storage, separately from primary-key managed BLOB compaction (#1369), 
primary-key vector indexes, or a new ANN indexing feature.
   
   ### Solution
   
   Port Java's 
[`DataEvolutionVectorCompactTask`](https://github.com/apache/paimon/blob/3821db1972b87d5cd3ac1d025ec214cc3842ae9a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionVectorCompactTask.java#L61-L275)
 and the [Vector planning 
rules](https://github.com/apache/paimon/blob/3821db1972b87d5cd3ac1d025ec214cc3842ae9a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java#L545-L613)
 into Rust's existing compaction pipeline.
   
   - Plan a complete set of Vector-store files for the selected normal row-ID 
group, including all overlapping versions and field projections. Only span 
adjacent normal groups when their normal files are compacted together in the 
same batch.
   - Resolve write columns using each file's schema and stable field IDs. For 
each current field represented in the inputs, require coverage of the entire 
task range. Skip unsafe groups with partial field coverage: filling missing 
coverage with NULL can overwrite values held elsewhere. Ignore dropped fields 
and do not materialize newly added fields absent from all inputs.
   - Read effective column values through the existing Data Evolution reader. 
Preserve field-wise version ordering: output files inherit the maximum input 
sequence, so omitting an overlapping input can change which values win.
   - Write the configured dedicated Vector format, rolling by 
`vector.target-file-size`. Assign consecutive first-row IDs to the outputs and 
verify that their combined range and physical row count exactly match the input 
union. Preserve sequence bounds and applicable per-column sequence metadata, 
including global-index column-update handling; validate winner selection again 
after subsequent field updates.
   - Integrate task serialization, output validation, snapshot publication, and 
failure cleanup. Preserve normal/BLOB files, logical deletions, and valid 
global indexes. Avoid repeatedly selecting already compacted rolled outputs 
when no consolidation is useful.
   
   #### Acceptance criteria
   
   - [ ] **Row-ID ranges:** test multiple rolling outputs, adjacent groups, 
overlapping versions, gaps, incomplete task inputs, and normal/Vector tasks 
committed together. Every physical slot retains its row ID, including deleted 
slots.
   - [ ] **Field versions:** cover multiple Vector fields updated 
independently, overlapping row ranges, NULL values, renames, dropped/recreated 
names, and newly added fields. Partial field coverage is skipped or rejected 
without publication; omitted versions cannot alter winners.
   - [ ] **Historical snapshots:** retained snapshots still return their 
original embeddings. Latest projected/full reads and existing 
vector-search/index paths return the same logical results after compaction.
   - [ ] **Commit/recovery:** test concurrent Vector/normal updates, disjoint 
appends, deletion-vector changes, retries, serialized task subsets, and 
abort/failure cleanup. Retained snapshots keep their referenced inputs.
   - [ ] Add Java-compatible fixtures or interoperability coverage, verify 
output format and size controls, show useful file consolidation, and confirm a 
second run does not endlessly rewrite completed outputs.
   - [ ] Update compaction documentation with supported configurations, 
invocation, and eligibility rules.
   
   ### Anything else?
   
   Suggested order: BLOB (#1380) first, then this Vector issue. Rough effort: 
several weeks for Vector, including field-coverage, compatibility, and recovery 
validation; this is a planning estimate.
   
   New ANN index formats, primary-key vector compaction, inline-to-dedicated 
migration, large normal-file splitting, and row-ID reassignment are outside 
this issue.
   
   Source baseline: paimon-rust `f91fdb8ab034887547a5ee2f26c1979e30250558`, 
compared with apache/paimon `3821db1972b87d5cd3ac1d025ec214cc3842ae9a`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to