SteNicholas opened a new pull request, #1421:
URL: https://github.com/apache/paimon-rust/pull/1421

   ### Purpose
   
   Linked issue: #1381
   
   Normal-file compaction leaves small dedicated Vector files and overlapping 
column-update versions in Data Evolution tables. This adds opt-in Vector 
storage compaction to the existing Rust compaction pipeline, following Java's 
`DataEvolutionVectorCompactTask` and coordinator planning rules.
   
   ### Brief change log
   
   - Plan dedicated Vector tasks by historical field ID, preserving renames and 
ignoring dropped fields or newly added fields absent from all inputs. Include 
every overlapping version and require complete coverage for each written field. 
Coordinate normal-file merges when a task crosses normal groups, and skip 
unchanged rolled outputs when no consolidation is useful.
   - Stream effective Vector values across schemas, projections and formats 
into dedicated outputs rolled by `vector.target-file-size`. Preserve NULLs, 
deleted physical slots, consecutive row IDs, file sequence bounds and column 
sequence metadata.
   - Revalidate complete input groups, historical field IDs and types, 
supported layouts and replacement metadata for local tasks and serialized 
results on every commit attempt. Reject omitted or concurrent overlapping 
Vector versions, partial field coverage and inline-to-dedicated or field-type 
migration.
   - Publish Vector and normal rewrites through the existing commit path, 
alongside optional BLOB compaction, preserving deletion vectors, global indexes 
and retained snapshots across retries and independently committed task subsets. 
Clean partial execution output and retain submitted files after failed or 
uncertain commits for recovery.
   
   ### Tests
   
   - Unit tests in `crates/paimon/src/table/compact/data_evolution/vector.rs`, 
`crates/paimon/src/table/data_evolution_reader/vector_versions_tests.rs` and 
the existing Data Evolution reader tests: overlapping groups, gaps, indexed 
range selection, useful rewrites, lazy reader reuse, exact stream lengths and 
late errors. Consolidate nullable and non-nullable missing-row coverage with 
row-range selection in one reader test.
   - Integration tests in 
`crates/paimon/tests/data_evolution_vector_compact_test.rs`: overlapping field 
versions, NULLs, rolling outputs, deleted slots, historical schemas and 
snapshots, renames and name reuse, partial coverage rejection, independent task 
subsets, concurrent updates, forged metadata, retry and failure cleanup, 
Java-written fixtures and mixed Parquet/Vortex formats. Verify a three-to-one 
Vector file reduction while preserving nonempty Top-4 search results, the 
expected nearest neighbor and valid global indexes.
   - Extend the existing Data Evolution storage-layout test to reject 
historical inline Vector providers, and run the existing normal-file and BLOB 
compaction suites against the shared planner and commit paths.
   
   - Validation: 237 related tests passed with the `vortex` feature: 177 Data 
Evolution unit tests and 60 integration tests across 
`data_evolution_vector_compact_test`, `blob_update_test`, 
`data_evolution_compact_test` and `data_evolution_compact_layout_test`. One 
existing manual benchmark was ignored. `cargo +1.98.0 fmt --all -- --check` and 
`cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex -- 
-D warnings` passed.
   
   ### API and Format
   
   Add `CompactBuilder::with_vector_compaction(bool)`, disabled by default, and 
an optional `vector_field_ids` in serialized compaction tasks. No storage 
format change. SQL, language bindings and automatic compaction continue to use 
normal-file compaction by default.
   
   ### Documentation
   
   Update `docs/src/compaction.md` with invocation, supported layouts, 
field-coverage requirements, output rolling, coordinated normal/BLOB/Vector 
commits, retry and cleanup semantics, and a file-count reduction example. 
Document Java fixture provenance in 
`crates/paimon/testdata/vector_compaction/README.md`.
   
   🤖 Assisted with Codex.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to