suxiaogang223 opened a new pull request, #1329:
URL: https://github.com/apache/paimon-rust/pull/1329

   ### Purpose
   
   Linked issue: Closes #1326.
   
   Related to #911 (partial scope).
   
   Repeated strings can retain several MiB of Arrow input while
   producing a small Parquet dictionary. Checking only encoded size can exhaust
   the input budget even when flushing the current file would allow writing to
   continue. This addresses one part of the resource-governance issue.
   
   Prepared with AI assistance. No throughput measurements are available.
   
   ### Brief change log
   
   Expose the current format writer's input reservation internally and include 
it
   in the data-file flush threshold. Reuse existing reservation release 
accounting;
   other files and readers sharing the context cannot trigger this file's flush.
   Writers without a resource context keep the encoded-size policy.
   
   ### Tests
   
   Validated with the existing Rust 1.97.0 toolchain on macOS arm64:
   
   - `cargo +1.97.0 test --locked -p paimon --lib`: 4804 passed, 6 ignored.
   - `cargo +1.97.0 test --locked -p paimon --test writer_resources_test`: 10 
passed.
   - New regressions cover four distinct approximately 4 MiB dictionary-friendly
     batches with a 6 MiB budget and a 1 MiB threshold, value readback and 
row-group
     boundaries, shared-context isolation, and reservation lifetime after failed
     flush/close. The existing auto-flush-with-tail test also checks per-file 
bytes.
   - `cargo +1.97.0 fmt --all -- --check` and `git diff --check`: passed.
   - `cargo +1.97.0 clippy --locked -p paimon --all-targets -- -D warnings`: 
passed.
   
   Not run locally: CI-pinned Rust 1.98.0 checks, the full workspace/Vortex
   suite, or the Linux/MSRV matrix. The results above cover the listed commands.
   
   ### API and Format
   
   Internal change; no public Rust or C API and no storage format change.
   Row-group boundaries can change with resource accounting. Formats whose flush
   is a no-op do not gain memory reclamation. Single-batch admission failures,
   background close overlap and allocations outside reservations remain 
unchanged.
   No spill, automatic retry or peak-memory guarantee is introduced.
   
   ### Documentation
   
   Document the per-file retained-input threshold in the C writer resource 
guide.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to