suxiaogang223 opened a new issue, #1326: URL: https://github.com/apache/paimon-rust/issues/1326
### Search before asking - [x] I searched the issues and pull requests and found no matching report for this specific flush condition. Related broader work: #911. ### Please describe the bug 🐞 The data-file writer uses encoded `in_progress_size` to decide when to flush. Repeated strings can retain several MiB of Arrow input while producing a small Parquet dictionary. With resource accounting enabled, input reservations can therefore exhaust the budget even when flushing the current file would allow writing to continue. A minimal case uses an append writer with a 6 MiB resource budget, a 1 MiB write-buffer threshold, and four separate batches of 1024 identical 4096-byte strings. Each batch must own distinct Arrow buffers, and file-size/row-count rolling should not trigger. Each batch fits the budget individually, but two retained batches do not. The encoded dictionary can remain below the flush threshold, causing a later write to fail with `ResourceExhausted`. Expected behavior: also consider the current file's retained-input reservation when applying the flush threshold, so these batches can be written and read back within their accounted budget. Source baseline: main `7a4f3bc89ad0d1ef814db82af1f5d2fba17fb12c`. The fix and regression tests were validated on macOS arm64 with Rust 1.97.0. ### Solution Expose the current format writer's own retained-input reservation internally and include it in the file flush decision. Reuse existing release accounting; do not use the shared ResourceContext total, which can include other writers or readers. Keep the encoded-size policy without a resource context. This addresses one part of #911. It does not add spill, retry after admission failure, background-close limits, or a process-memory bound. Prepared with AI assistance; no throughput measurements are available. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
