suxiaogang223 opened a new pull request, #1329:
URL: https://github.com/apache/paimon-rust/pull/1329
### Purpose
Linked issue: Closes #1326.
Related to #911 (partial scope).
Repeated strings can retain several MiB of Arrow input while
producing a small Parquet dictionary. Checking only encoded size can exhaust
the input budget even when flushing the current file would allow writing to
continue. This addresses one part of the resource-governance issue.
Prepared with AI assistance. No throughput measurements are available.
### Brief change log
Expose the current format writer's input reservation internally and include
it
in the data-file flush threshold. Reuse existing reservation release
accounting;
other files and readers sharing the context cannot trigger this file's flush.
Writers without a resource context keep the encoded-size policy.
### Tests
Validated with the existing Rust 1.97.0 toolchain on macOS arm64:
- `cargo +1.97.0 test --locked -p paimon --lib`: 4804 passed, 6 ignored.
- `cargo +1.97.0 test --locked -p paimon --test writer_resources_test`: 10
passed.
- New regressions cover four distinct approximately 4 MiB dictionary-friendly
batches with a 6 MiB budget and a 1 MiB threshold, value readback and
row-group
boundaries, shared-context isolation, and reservation lifetime after failed
flush/close. The existing auto-flush-with-tail test also checks per-file
bytes.
- `cargo +1.97.0 fmt --all -- --check` and `git diff --check`: passed.
- `cargo +1.97.0 clippy --locked -p paimon --all-targets -- -D warnings`:
passed.
Not run locally: CI-pinned Rust 1.98.0 checks, the full workspace/Vortex
suite, or the Linux/MSRV matrix. The results above cover the listed commands.
### API and Format
Internal change; no public Rust or C API and no storage format change.
Row-group boundaries can change with resource accounting. Formats whose flush
is a no-op do not gain memory reclamation. Single-batch admission failures,
background close overlap and allocations outside reservations remain
unchanged.
No spill, automatic retry or peak-memory guarantee is introduced.
### Documentation
Document the per-file retained-input threshold in the C writer resource
guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]