suxiaogang223 opened a new pull request, #66810: URL: https://github.com/apache/doris/pull/66810
## What changed - partition each JNI writer's managed memory budget between the Paimon page pool and a finite Java Arrow allocator - split large Doris Blocks into row-range Arrow IPC batches and preallocate IPC buffers to reduce transient amplification - reuse row-conversion scratch storage and avoid pinning Java commit payload arrays during JNI extraction - classify Arrow, Paimon page, JVM heap, C++ Arrow, and commit-payload memory failures, expose profile counters, and fail the current query with a memory error - cap the aggregate serialized commit payload using the inactive Arrow transient headroom during commit ## Why The Paimon page pool was bounded, but Java Arrow decoding used an effectively unlimited allocator and large Blocks could temporarily coexist in several native/direct representations. Under concurrency, these independent allocations could create large memory spikes or JVM-level OOMs that risk affecting unrelated queries. This change establishes per-writer hard boundaries, keeps normal pressure on Paimon's existing preempt/spill/flush path, reduces transient copies, and converts unrecoverable memory pressure into a query-local failure. ## Impact Paimon JNI writes now have bounded Arrow direct memory and smaller conversion batches. Insufficient memory is handled by existing queue backpressure and Paimon spill/flush where possible; otherwise the current query receives a categorized memory error. Bucket routing, file layout, and commit semantics are unchanged. ## Validation - `git diff --check` - BE source formatting verified with `clang-format --dry-run --Werror` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
