SteNicholas opened a new pull request, #1407: URL: https://github.com/apache/paimon-rust/pull/1407
### Purpose Linked issue: #1380 Normal-file compaction leaves small dedicated BLOB files and overlapping column-update versions in Data Evolution tables. This adds opt-in BLOB payload compaction to the existing Rust compaction pipeline, following Java's `DataEvolutionBlobCompactTask` and coordinator planning rules. ### Brief change log - Plan dedicated BLOB tasks by historical field ID, preserving renames and keeping dropped/recreated columns with reused names distinct. Keep overlapping versions together and pack contiguous ranges by `blob.target-file-size`, with a two-file minimum and coordinated normal-file merges when a task crosses normal groups. - Stream effective `BLOB`, `ARRAY<BLOB>` and `MAP<K, BLOB>` values into one dedicated output per task. Resolve sparse-update placeholders through older versions and preserve NULLs, deleted physical slots, row IDs, file sequence bounds and column sequence metadata. - Revalidate complete input groups, historical field IDs and types, supported layouts and replacement metadata for local tasks and serialized results on every commit attempt. Reject concurrent overlapping versions of the same BLOB field, video layouts and inline-to-dedicated or field-type migration. - Publish normal and BLOB rewrites through the existing commit path, preserving deletion vectors, global indexes and retained snapshots across retries and independently committed task subsets. Clean partial execution output and retain submitted files after failed or uncertain commits for recovery. ### Tests - Unit tests in `crates/paimon/src/table/compact/data_evolution/blob.rs` and `crates/paimon/src/table/table_commit/parity_tests.rs`: Java size boundaries, overlapping groups, gaps and invalid ranges, concurrent updates and deletions during commit retries, uncertain publication and idempotent recovery. - Integration tests in `crates/paimon/tests/blob_update/compaction.rs`: scalar and collection BLOB values, sparse updates, NULLs and fallbacks, historical schemas, renames and name reuse, independent task subsets, concurrent writes to other BLOB fields, forged metadata, failure cleanup, retained snapshots and Java-written fixtures. Verify a three-to-one BLOB file reduction without changing the five physical row slots. - Extend the existing Data Evolution compaction and storage-layout tests to cover opt-in BLOB rewrites, unchanged Vector files and rejection of historical inline providers. Combine collection update and compaction coverage in one test. - Validation after rebasing: all 43 tests in `blob_update_test`, `data_evolution_compact_test` and `data_evolution_compact_layout_test`, plus `cargo fmt --all -- --check`, passed. Before rebasing, the full core run passed 5,788 tests with 6 ignored, and workspace Clippy passed with `-D warnings`. Local `--all-targets` validation is not fully green because macOS terminates the empty `rest_catalog_example` test harness with SIGKILL, including after rebuilding it. ### API and Format Add `CompactBuilder::with_blob_compaction(bool)`, disabled by default, and an optional `blob_field_id` in serialized compaction tasks. No storage format change. SQL, language bindings and automatic compaction continue to use normal-file compaction by default. ### Documentation Update `docs/src/compaction.md` with invocation, supported layouts, task sizing, coordinated normal/BLOB commits, retry and cleanup semantics, and a file-count reduction example. 🤖 Assisted with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
