SteNicholas opened a new issue, #1380: URL: https://github.com/apache/paimon-rust/issues/1380
### Search before asking - [x] I searched existing issues and pull requests and found no issue tracking dedicated Data Evolution BLOB file compaction. ### Motivation Multimodal Data Evolution tables accumulate small dedicated BLOB files and overlapping column-update versions. Rust's normal-column compaction preserves those files, so it cannot consolidate their payload storage or reduce the associated file-open and metadata overhead. The existing Data Evolution implementation (#1249) supports normal files only; [the documented limitation](https://github.com/apache/paimon-rust/blob/f91fdb8ab034887547a5ee2f26c1979e30250558/docs/src/compaction.md#L647-L699) explicitly excludes dedicated BLOB/Vector compaction. This issue concerns append tables with `data-evolution.enabled=true`, `row-tracking.enabled=true`, `bucket=-1`, and no primary key. They reconstruct columns by row-ID range and field version. This is a separate mechanism from [primary-key managed BLOB compaction (#1369)](https://github.com/apache/paimon-rust/pull/1369), which merges descriptors and rebuilds `.blobref` sidecars without rewriting payload packs. ### Solution Port Java's [`DataEvolutionBlobCompactTask`](https://github.com/apache/paimon/blob/3821db1972b87d5cd3ac1d025ec214cc3842ae9a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionBlobCompactTask.java#L57-L196) and the BLOB planning rules in [`DataEvolutionCompactCoordinator`](https://github.com/apache/paimon/blob/3821db1972b87d5cd3ac1d025ec214cc3842ae9a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java#L639-L728) into the existing Rust compaction pipeline. - Add BLOB task planning, execution, serialization, and commit validation alongside normal tasks. Reuse snapshot capture, task subsets, retries, and output cleanup. - Resolve each input's single write column through its historical schema and group by stable field ID. Handle renames correctly and keep dropped/recreated columns with the same name distinct. Only compact fields still configured for dedicated BLOB storage. - Keep overlapping versions together and form contiguous row-ID ranges. Follow Java's dedicated-file association with normal row groups, two-file minimum, and `blob.target-file-size` selection. Consolidation across normal groups must respect coordinated normal-file rewrites; never bridge a row-ID gap. - Read the effective BLOB values in row-ID order using the existing version/fallback rules and stream them into one dedicated output per task, as Java does. Cover scalar BLOB and the collection forms already supported by the Blob format; explicitly reject unsupported dedicated layouts. - Preserve the exact physical row-ID range, row count, input sequence bounds, write-column identity, and applicable column sequence metadata without changing field-version selection. Retain deleted slots and logical deletions. Keep normal-file/DV handling and unrelated fields correct when normal and BLOB tasks share a commit. #### Acceptance criteria - [ ] **Row-ID ranges:** adjacent segments and overlapping versions compact correctly; gaps, invalid ranges, and incomplete input groups cannot publish. Verify independent BLOB tasks and combined normal/BLOB commits. - [ ] **Field versions:** validate partial and sparse BLOB updates, NULL versus fallback values, multiple BLOB fields, historical schemas, renames, and name reuse. Logical values and latest-version selection remain correct after compaction and subsequent updates. - [ ] **Historical snapshots:** retained snapshots still read their original values and payloads. Current reads preserve logical deletions, unaffected Vector files, and valid global-index state. - [ ] **Commit/recovery:** revalidate relevant inputs against concurrent updates; test commit retries, serialized task subsets, abort/failure cleanup, and idempotent recovery without deleting files referenced by retained snapshots. - [ ] Add Java-compatible fixtures or interoperability coverage, document supported layouts and invocation, and show file-count reduction on a representative multimodal workload. ### Anything else? Suggested order: implement BLOB first, then dedicated Vector compaction in a separate issue. Rough effort: several weeks for BLOB, including compatibility and recovery validation; this is a planning estimate. Vector compaction, primary-key managed payload-pack rewriting, inline-to-dedicated migration, large normal-file splitting, and row-ID reassignment are outside this issue. Source baseline: paimon-rust `f91fdb8ab034887547a5ee2f26c1979e30250558`, compared with apache/paimon `3821db1972b87d5cd3ac1d025ec214cc3842ae9a`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
