SteNicholas opened a new pull request, #1407:
URL: https://github.com/apache/paimon-rust/pull/1407

   ### Purpose
   
   Linked issue: #1380
   
   Normal-file compaction leaves small dedicated BLOB files and overlapping 
column-update versions in Data Evolution tables. This adds opt-in BLOB payload 
compaction to the existing Rust compaction pipeline, following Java's 
`DataEvolutionBlobCompactTask` and coordinator planning rules.
   
   ### Brief change log
   
   - Plan dedicated BLOB tasks by historical field ID, preserving renames and 
keeping dropped/recreated columns with reused names distinct. Keep overlapping 
versions together and pack contiguous ranges by `blob.target-file-size`, with a 
two-file minimum and coordinated normal-file merges when a task crosses normal 
groups.
   - Stream effective `BLOB`, `ARRAY<BLOB>` and `MAP<K, BLOB>` values into one 
dedicated output per task. Resolve sparse-update placeholders through older 
versions and preserve NULLs, deleted physical slots, row IDs, file sequence 
bounds and column sequence metadata.
   - Revalidate complete input groups, historical field IDs and types, 
supported layouts and replacement metadata for local tasks and serialized 
results on every commit attempt. Reject concurrent overlapping versions of the 
same BLOB field, video layouts and inline-to-dedicated or field-type migration.
   - Publish normal and BLOB rewrites through the existing commit path, 
preserving deletion vectors, global indexes and retained snapshots across 
retries and independently committed task subsets. Clean partial execution 
output and retain submitted files after failed or uncertain commits for 
recovery.
   
   ### Tests
   
   - Unit tests in `crates/paimon/src/table/compact/data_evolution/blob.rs` and 
`crates/paimon/src/table/table_commit/parity_tests.rs`: Java size boundaries, 
overlapping groups, gaps and invalid ranges, concurrent updates and deletions 
during commit retries, uncertain publication and idempotent recovery.
   - Integration tests in `crates/paimon/tests/blob_update/compaction.rs`: 
scalar and collection BLOB values, sparse updates, NULLs and fallbacks, 
historical schemas, renames and name reuse, independent task subsets, 
concurrent writes to other BLOB fields, forged metadata, failure cleanup, 
retained snapshots and Java-written fixtures. Verify a three-to-one BLOB file 
reduction without changing the five physical row slots.
   - Extend the existing Data Evolution compaction and storage-layout tests to 
cover opt-in BLOB rewrites, unchanged Vector files and rejection of historical 
inline providers. Combine collection update and compaction coverage in one test.
   
   - Validation after rebasing: all 43 tests in `blob_update_test`, 
`data_evolution_compact_test` and `data_evolution_compact_layout_test`, plus 
`cargo fmt --all -- --check`, passed. Before rebasing, the full core run passed 
5,788 tests with 6 ignored, and workspace Clippy passed with `-D warnings`. 
Local `--all-targets` validation is not fully green because macOS terminates 
the empty `rest_catalog_example` test harness with SIGKILL, including after 
rebuilding it.
   
   ### API and Format
   
   Add `CompactBuilder::with_blob_compaction(bool)`, disabled by default, and 
an optional `blob_field_id` in serialized compaction tasks. No storage format 
change. SQL, language bindings and automatic compaction continue to use 
normal-file compaction by default.
   
   ### Documentation
   
   Update `docs/src/compaction.md` with invocation, supported layouts, task 
sizing, coordinated normal/BLOB commits, retry and cleanup semantics, and a 
file-count reduction example.
   
   🤖 Assisted with Codex.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to