jianguotian opened a new pull request, #854:
URL: https://github.com/apache/paimon-rust/pull/854
## Purpose
Add the common manifest-sorting engine before reintroducing bucket-first
sorting. This follows Java Paimon's `ManifestFileSorter` foundation and avoids
the bucket-only write-path special case that was closed in #852.
## Changes
- Add the Java-compatible manifest-sort options and eager schema validation:
- `manifest-sort.enabled`
- `manifest-sort.partition-field`
- `manifest-sort.max-rewrite-size`
- `manifest.full-compaction-threshold-size`
- universal-compaction size amplification and ratio options
- Add partition-key sorting for manifest entries.
- Build non-overlapping sorted runs, select runs with the
universal-compaction strategy, and split overlapping rewrite sections.
- Apply the sorted path to ordinary minor and full manifest compaction while
preserving the legacy path when sorting is disabled.
- Preserve ADD/DELETE semantics in both paths, including unmatched DELETE
entries in minor compaction.
- Enforce the rewrite budget at manifest-file granularity and guarantee
progress by admitting the first crossing file and at least two files.
- Add bounded external sorting:
- spill after `sort-spill-buffer-size`
- bound merge fan-in with `local-sort.max-num-file-handles`
- enforce `write-buffer-spill.max-disk-size`
- clean owned spill files through the temporary-directory lifetime
- Keep output manifest writing bounded by the manifest target size.
## Scope
This PR intentionally establishes only the shared partition-sort engine.
Follow-up PRs will add, in merge order:
1. Data Evolution RowID sort key
2. bucket-first sort key for fixed/postponed buckets
3. `compact_manifest` maintenance API
4. explicit full-sort behavior
## Verification
- `cargo fmt --check`
- `cargo test -p paimon --lib` — 2,842 passed, 2 ignored
- `cargo clippy -p paimon --lib --tests -- -D warnings -A
clippy::unnecessary-cast`
- the allowance is for an unrelated existing Rust 1.95 lint in
`lumina/ffi.rs`
- dedicated coverage for run planning, minor/full DELETE handling, spill
ordering, fan-in reduction, disk limits, options, and schema validation
## Related Java work
- apache/paimon#7842
- apache/paimon#8119
- apache/paimon#8357
- apache/paimon#8422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
