suxiaogang223 opened a new issue, #911: URL: https://github.com/apache/paimon-rust/issues/911
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation Native query engines need a Paimon writer to run within the query's resource limits. Today the Rust writer and C FFI do not provide a way to connect the write path to an engine memory tracker, spill memory-heavy work to disk, stop promptly when a query is cancelled, or report writer resource metrics. The existing `write_buffer_size` option is a flush threshold, not a total writer memory limit. Writer memory also includes: - Arrow `RecordBatch` values retained by primary-key writers; - active partition and bucket writers; - file encoder and row-group buffers; - sorting, deduplication, merging, serialization, and index-generation state; - file writers being closed by background tasks. Changing `write_buffer_size` cannot account for all of this memory or reserve bytes from the embedding query engine before an allocation. An engine such as Apache Doris therefore cannot reliably enforce its query memory limit around a Paimon Rust write. The C FFI also has no cancellation handle or writer metrics API. A cancelled query may continue CPU work or I/O until the current operation finishes, and the engine cannot report the memory retained or data spilled by the writer. ### Solution This requires changes in both the Rust writer core and the C FFI. FFI callbacks alone are insufficient because the allocations and long-running work occur inside the Rust writer. #### Rust writer core Add an optional resource context shared by all writers in one logical write. Existing callers should retain the current behavior when no context is configured. The context should support: - reserving memory before retaining buffers or starting memory-heavy work; - automatically releasing reservations when the memory is freed; - returning a distinct resource-exhausted error when memory cannot be reserved; - checking a cancellation signal; - configuring a spill directory and spill budget; - collecting writer metrics. Memory accounting should initially cover retained Arrow batches, active file/row-group buffers, primary-key sort and merge state, index state, and background file-close tasks. Reservations should use an ownership or guard model so error and cancellation paths release them reliably. When memory cannot be reserved, the writer should flush safe state or spill intermediate data. Primary-key sorting, deduplication, and merging need a bounded-memory spill path rather than only a lower file flush threshold. If an operation cannot spill safely, it should return a resource-exhausted error. Spill files should be scoped to one writer or query and removed after success, error, abort, or cancellation. The spilled path must produce the same logical result as the in-memory path. Long-running operations should check cancellation at useful boundaries, including batch ingestion, partition/bucket dispatch, sort and merge loops, file encoding, background close waits, and `prepare_commit`. Cancellation should return a distinct error and clean up reservations, tasks, open writers, and spill files without publishing a commit. Expose a metrics snapshot containing at least: - current and peak accounted memory; - input rows and bytes; - output files and bytes; - spill operations, files, and bytes; - flush count and time; - sort/merge and `prepare_commit` time; - cancellation and resource-exhausted counts. The API should document which values are exact and which are estimates. #### C FFI Expose the resource context through a versioned C structure or an opaque thread-safe handle. An embedding engine needs to: - provide memory reserve/release callbacks or a fixed memory budget; - provide spill configuration; - cancel a writer while another FFI call may be running; - retrieve a metrics snapshot; - distinguish cancellation and resource exhaustion from other errors. A possible callback shape is: ```c typedef struct paimon_write_resource_callbacks_v1 { void *context; bool (*try_reserve)(void *context, uint64_t bytes); void (*release)(void *context, uint64_t bytes); bool (*is_cancelled)(void *context); } paimon_write_resource_callbacks_v1; ``` The final design must define callback lifetime, thread-safety, ownership, and whether callbacks can be invoked from Rust runtime threads. A separate resource handle may be preferable because cancellation must work while a write call is blocked in the FFI. #### Suggested implementation phases 1. Add the resource context, memory reservations, cancellation, and metrics to the Rust core. 2. Add the C FFI resource handle and error/metrics APIs. 3. Add bounded-memory primary-key sort/merge with spill. 4. Extend accounting and spill coverage to the remaining writer and index paths. The implementation should demonstrate that: - a configured writer stays within its accounted budget or returns a resource-exhausted error; - a low-memory primary-key write spills and produces the same logical result as the in-memory path; - cancellation stops a long-running write and removes reservations, tasks, and spill files; - an embedding engine can attach its query memory tracker and read writer metrics. ### Anything else? The closest existing issues found during the search are: - #665 reduces scan-planning memory use; - #673 configures DataFusion runtime resources; - #679 bounds memory used for long snapshot histories. These issues concern scan, SQL runtime, or metadata paths rather than table writers. Open design questions: 1. Should the core abstraction expose reserve/release callbacks, a Rust memory-pool trait, or both? 2. Which writer allocations can be accounted exactly in the first phase? 3. Should spill be limited to primary-key sort/merge initially? 4. What cancellation latency should the API guarantee? ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
