SteNicholas opened a new pull request, #1421: URL: https://github.com/apache/paimon-rust/pull/1421
### Purpose Linked issue: #1381 Normal-file compaction leaves small dedicated Vector files and overlapping column-update versions in Data Evolution tables. This adds opt-in Vector storage compaction to the existing Rust compaction pipeline, following Java's `DataEvolutionVectorCompactTask` and coordinator planning rules. ### Brief change log - Plan dedicated Vector tasks by historical field ID, preserving renames and ignoring dropped fields or newly added fields absent from all inputs. Include every overlapping version and require complete coverage for each written field. Coordinate normal-file merges when a task crosses normal groups, and skip unchanged rolled outputs when no consolidation is useful. - Stream effective Vector values across schemas, projections and formats into dedicated outputs rolled by `vector.target-file-size`. Preserve NULLs, deleted physical slots, consecutive row IDs, file sequence bounds and column sequence metadata. - Revalidate complete input groups, historical field IDs and types, supported layouts and replacement metadata for local tasks and serialized results on every commit attempt. Reject omitted or concurrent overlapping Vector versions, partial field coverage and inline-to-dedicated or field-type migration. - Publish Vector and normal rewrites through the existing commit path, alongside optional BLOB compaction, preserving deletion vectors, global indexes and retained snapshots across retries and independently committed task subsets. Clean partial execution output and retain submitted files after failed or uncertain commits for recovery. ### Tests - Unit tests in `crates/paimon/src/table/compact/data_evolution/vector.rs`, `crates/paimon/src/table/data_evolution_reader/vector_versions_tests.rs` and the existing Data Evolution reader tests: overlapping groups, gaps, indexed range selection, useful rewrites, lazy reader reuse, exact stream lengths and late errors. Consolidate nullable and non-nullable missing-row coverage with row-range selection in one reader test. - Integration tests in `crates/paimon/tests/data_evolution_vector_compact_test.rs`: overlapping field versions, NULLs, rolling outputs, deleted slots, historical schemas and snapshots, renames and name reuse, partial coverage rejection, independent task subsets, concurrent updates, forged metadata, retry and failure cleanup, Java-written fixtures and mixed Parquet/Vortex formats. Verify a three-to-one Vector file reduction while preserving nonempty Top-4 search results, the expected nearest neighbor and valid global indexes. - Extend the existing Data Evolution storage-layout test to reject historical inline Vector providers, and run the existing normal-file and BLOB compaction suites against the shared planner and commit paths. - Validation: 237 related tests passed with the `vortex` feature: 177 Data Evolution unit tests and 60 integration tests across `data_evolution_vector_compact_test`, `blob_update_test`, `data_evolution_compact_test` and `data_evolution_compact_layout_test`. One existing manual benchmark was ignored. `cargo +1.98.0 fmt --all -- --check` and `cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex -- -D warnings` passed. ### API and Format Add `CompactBuilder::with_vector_compaction(bool)`, disabled by default, and an optional `vector_field_ids` in serialized compaction tasks. No storage format change. SQL, language bindings and automatic compaction continue to use normal-file compaction by default. ### Documentation Update `docs/src/compaction.md` with invocation, supported layouts, field-coverage requirements, output rolling, coordinated normal/BLOB/Vector commits, retry and cleanup semantics, and a file-count reduction example. Document Java fixture provenance in `crates/paimon/testdata/vector_compaction/README.md`. 🤖 Assisted with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
