SteNicholas opened a new issue, #1380:
URL: https://github.com/apache/paimon-rust/issues/1380

   ### Search before asking
   
   - [x] I searched existing issues and pull requests and found no issue 
tracking dedicated Data Evolution BLOB file compaction.
   
   ### Motivation
   
   Multimodal Data Evolution tables accumulate small dedicated BLOB files and 
overlapping column-update versions. Rust's normal-column compaction preserves 
those files, so it cannot consolidate their payload storage or reduce the 
associated file-open and metadata overhead.
   
   The existing Data Evolution implementation (#1249) supports normal files 
only; [the documented 
limitation](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/docs/src/compaction.md#L647-L699)
 explicitly excludes dedicated BLOB/Vector compaction.
   
   This issue concerns append tables with `data-evolution.enabled=true`, 
`row-tracking.enabled=true`, `bucket=-1`, and no primary key. They reconstruct 
columns by row-ID range and field version. This is a separate mechanism from 
[primary-key managed BLOB compaction 
(#1369)](https://github.com/apache/paimon-rust/pull/1369), which merges 
descriptors and rebuilds `.blobref` sidecars without rewriting payload packs.
   
   ### Solution
   
   Port Java's 
[`DataEvolutionBlobCompactTask`](https://github.com/apache/paimon/blob/3821db1972b87d5cd3ac1d025ec214cc3842ae9a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionBlobCompactTask.java#L57-L196)
 and the BLOB planning rules in 
[`DataEvolutionCompactCoordinator`](https://github.com/apache/paimon/blob/3821db1972b87d5cd3ac1d025ec214cc3842ae9a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java#L639-L728)
 into the existing Rust compaction pipeline.
   
   - Add BLOB task planning, execution, serialization, and commit validation 
alongside normal tasks. Reuse snapshot capture, task subsets, retries, and 
output cleanup.
   - Resolve each input's single write column through its historical schema and 
group by stable field ID. Handle renames correctly and keep dropped/recreated 
columns with the same name distinct. Only compact fields still configured for 
dedicated BLOB storage.
   - Keep overlapping versions together and form contiguous row-ID ranges. 
Follow Java's dedicated-file association with normal row groups, two-file 
minimum, and `blob.target-file-size` selection. Consolidation across normal 
groups must respect coordinated normal-file rewrites; never bridge a row-ID gap.
   - Read the effective BLOB values in row-ID order using the existing 
version/fallback rules and stream them into one dedicated output per task, as 
Java does. Cover scalar BLOB and the collection forms already supported by the 
Blob format; explicitly reject unsupported dedicated layouts.
   - Preserve the exact physical row-ID range, row count, input sequence 
bounds, write-column identity, and applicable column sequence metadata without 
changing field-version selection. Retain deleted slots and logical deletions. 
Keep normal-file/DV handling and unrelated fields correct when normal and BLOB 
tasks share a commit.
   
   #### Acceptance criteria
   
   - [ ] **Row-ID ranges:** adjacent segments and overlapping versions compact 
correctly; gaps, invalid ranges, and incomplete input groups cannot publish. 
Verify independent BLOB tasks and combined normal/BLOB commits.
   - [ ] **Field versions:** validate partial and sparse BLOB updates, NULL 
versus fallback values, multiple BLOB fields, historical schemas, renames, and 
name reuse. Logical values and latest-version selection remain correct after 
compaction and subsequent updates.
   - [ ] **Historical snapshots:** retained snapshots still read their original 
values and payloads. Current reads preserve logical deletions, unaffected 
Vector files, and valid global-index state.
   - [ ] **Commit/recovery:** revalidate relevant inputs against concurrent 
updates; test commit retries, serialized task subsets, abort/failure cleanup, 
and idempotent recovery without deleting files referenced by retained snapshots.
   - [ ] Add Java-compatible fixtures or interoperability coverage, document 
supported layouts and invocation, and show file-count reduction on a 
representative multimodal workload.
   
   ### Anything else?
   
   Suggested order: implement BLOB first, then dedicated Vector compaction in a 
separate issue. Rough effort: several weeks for BLOB, including compatibility 
and recovery validation; this is a planning estimate.
   
   Vector compaction, primary-key managed payload-pack rewriting, 
inline-to-dedicated migration, large normal-file splitting, and row-ID 
reassignment are outside this issue.
   
   Source baseline: paimon-rust `f91fdb8ab034887547a5ee2f26c1979e30250558`, 
compared with apache/paimon `3821db1972b87d5cd3ac1d025ec214cc3842ae9a`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to