JingsongLi commented on issue #911: URL: https://github.com/apache/paimon-rust/issues/911#issuecomment-5773332474
I've opened #914 as an initial step toward a common resource-management API. The proposal is to validate the shared memory-reservation and ownership model in the Reader layer first, then extend the same foundation to the Writer. The current design discussion focuses on Rust and Parquet; C FFI integration can follow once the core contracts are clearer. The API in #914 consists of: - `ResourceContext`: a shared budget and current/peak reservation metrics, passed through `ReadBuilder::with_resources` or `TableRead::with_resources`. - `MemoryPool::try_reserve(bytes)` / `release(bytes)`: an adapter for an embedding engine's memory budget. Failed reservations leave accounting unchanged; callbacks may run on Rust runtime threads. - `MemoryReservation`: an ownership guard that returns the reservation on drop. Reader output reservations follow the shared Arrow buffer owners, including cloned arrays and slices. This first PR accounts for retained output buffers before emitting them. Decoder working memory, prefetch buffers, and merge state are still outside that coverage, so it does not yet establish a complete Reader memory bound or resolve the Writer requirements in this issue. Existing `ReadBudget` remains in place. A subsequent step can move its byte accounting onto the common reservation mechanism while retaining separate concurrency and prefetch controls. **We would particularly appreciate a review from Doris maintainers or contributors familiar with its memory-management design.** Could a Doris expert compare this interface with Doris's existing design and assess whether it is a suitable integration boundary? In particular: 1. How should these reservations connect to Doris's query memory tracking and reservation model, including avoiding double accounting when Arrow buffers cross the engine/library boundary? 2. Are fallible, non-blocking reservation and infallible release callbacks sufficient, or is an explicit contract needed for coordinating memory reclamation, flush, and spill? Who should initiate that work when admission fails? 3. What execution-context and lifetime guarantees are needed when reservations are acquired or released by asynchronous Rust tasks, or when output buffers outlive the reader? Feedback on these points would help us settle the common interface before extending accounting deeper into Parquet readers and implementing Writer resource governance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
