suxiaogang223 opened a new issue, #1326:
URL: https://github.com/apache/paimon-rust/issues/1326

   ### Search before asking
   
   - [x] I searched the issues and pull requests and found no matching report 
for this specific flush condition. Related broader work: #911.
   
   ### Please describe the bug 🐞
   
   The data-file writer uses encoded `in_progress_size` to decide when to flush.
   Repeated strings can retain several MiB of Arrow input while producing a 
small
   Parquet dictionary. With resource accounting enabled, input reservations can
   therefore exhaust the budget even when flushing the current file would allow
   writing to continue.
   
   A minimal case uses an append writer with a 6 MiB resource budget, a 1 MiB
   write-buffer threshold, and four separate batches of 1024 identical 4096-byte
   strings. Each batch must own distinct Arrow buffers, and file-size/row-count
   rolling should not trigger. Each batch fits the budget individually, but two
   retained batches do not. The encoded dictionary can remain below the flush
   threshold, causing a later write to fail with `ResourceExhausted`.
   
   Expected behavior: also consider the current file's retained-input 
reservation
   when applying the flush threshold, so these batches can be written and read
   back within their accounted budget.
   
   Source baseline: main `7a4f3bc89ad0d1ef814db82af1f5d2fba17fb12c`.
   The fix and regression tests were validated on macOS arm64 with Rust 1.97.0.
   
   ### Solution
   
   Expose the current format writer's own retained-input reservation internally
   and include it in the file flush decision. Reuse existing release accounting;
   do not use the shared ResourceContext total, which can include other writers
   or readers. Keep the encoded-size policy without a resource context.
   
   This addresses one part of #911. It does not add spill, retry after admission
   failure, background-close limits, or a process-memory bound. Prepared with AI
   assistance; no throughput measurements are available.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to