JingsongLi opened a new pull request, #1264:
URL: https://github.com/apache/paimon-rust/pull/1264
### Purpose
Follow-up to #1261. Primary-key compaction previously rejected source-backed
index manifests or rewrote data without building declared PK indexes. Maintain
those indexes with the compacted data using Java's PK index options and source
policy.
### Brief change log
- Resolve `pk-{btree,bitmap,multivalue,fm,full-text,vector}.index.columns`
and per-field JSON options. Reuse the native writers, spill sorter, physical
file reader and vector training path.
- Build one payload per partition, bucket, positive level and column from
the complete compacted file set in filename order, retaining physical
NULL/tombstone positions.
- Carry index additions and retirements in `CompactResult` and publish them
atomically with data, changelogs and deletion vectors. Revalidate captured
index membership and complete source sets on every commit attempt while
preserving unrelated indexes and concurrent L0 appends.
- Clean private outputs on build failure, retain prepared outputs on
failed/uncertain publication, and recover duplicate legacy results before
checking expired sources. Replace incompatible vector definitions atomically.
- Share field-option parsing with reads and protect declared scalar index
columns from destructive schema changes, matching Java.
Declarations are unset by default. Automatic compaction maintains them with
`write-only=false`; explicit compaction also maintains them with
`write-only=true`.
### Tests
- Added 15 PK compaction integration tests covering native families,
physical positions, incremental/history retention, HASH/DV coexistence,
independent buckets, conflicting sources/indexes, schema/config changes,
callback recovery, missing/tampered outputs and injected stat failure cleanup.
- Added direct native scalar-payload assertions, Java source-metadata
serialization checks, scalar indexed-column DDL guards and declared/query
dimension precedence.
- `cargo test --locked -p paimon --all-targets --features vortex --
--test-threads=8`
- `cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex
-- -D warnings`
- `cargo +1.98.0 fmt --all -- --check`
The initial default-parallel run hit the existing shared-cache child's
five-second startup timeout. Its isolated rerun passed; the complete run above
uses bounded parallelism.
### API and Format
`CompactResult` adds optional serialized `index_entries` and an accessor.
Older results still deserialize: first publication requires re-execution when
index maintenance is missing; already published results remain idempotently
recoverable.
PK source metadata uses Java's existing version-1 DataOutput/writeUTF
encoding. Native index formats and existing PK read constraints are unchanged.
### Documentation
Updated `docs/src/compaction.md` with PK declarations, maintenance units,
atomicity, concurrency and recovery behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
