fengruotj opened a new issue, #11052:
URL: https://github.com/apache/paimon/issues/11052

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
   Paimon exposes `sink.writer-cpu`, `sink.writer-memory`, and 
`sink.writer-managed-memory` to configure the writer slot sharing group's CPU, 
task heap, and managed memory resources. There is currently no corresponding 
option for task off-heap memory.
   
   When the writer uses an explicit resource profile, its task off-heap request 
remains zero. Workloads that need native or direct memory on the write path 
therefore cannot express that requirement through Paimon's writer resource 
options. Increasing task heap or managed memory does not configure the task 
off-heap component.
   
   For example, a workload may need 512 MiB of task heap, 128 MiB of task 
off-heap memory, and 64 MiB of managed memory per writer shared slot. Exposing 
the missing resource dimension would let users declare these requirements 
independently in deployments that support Flink's fine-grained resource 
management.
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   Introduce `sink.writer-offheap-memory` as an optional `MemorySize` table 
property without a default. Pass its byte value to 
`SlotSharingGroup.Builder.setTaskOffHeapMemory()` when constructing the writer 
group.
   
   Example:
   
   ```properties
   sink.writer-memory = 512 mb
   sink.writer-offheap-memory = 128 mb
   sink.writer-managed-memory = 64 mb
   ```
   
   A local Flink 1.20.1 probe exercised the sink's shared 
resource-configuration helper and checked both StreamGraph and JobGraph 
profiles:
   
   ```text
                                         CPU    Task heap    Task off-heap    
Managed
   Option absent                          1      512 MiB        0 MiB         
64 MiB
   sink.writer-offheap-memory = 128 mb    1      512 MiB      128 MiB         
64 MiB
   ```
   
   The option describes the resource requirement of each shared slot in the 
writer group. It does not reserve a total amount across all writer subtasks, 
guarantee exclusive memory for a writer, or enforce a per-writer native-memory 
limit. Actual allocation depends on Flink's scheduler and available resources.
   
   The proposal has the following behavior:
   
   - Require `sink.writer-memory` and a positive off-heap value. Continue using 
the existing default of one CPU core unless `sink.writer-cpu` is configured.
   - Accept memory units such as `kb`, `mb`, and `gb`, preserving byte 
precision. For example, `1536 kb` produces a request of 1,572,864 bytes.
   - Keep task off-heap and managed memory independent. The option does not 
change writer-buffer allocation, enable `sink.use-managed-memory-allocator`, or 
alter `sink.managed.writer-buffer-memory` weights.
   - Support both regular Flink sink writers and combined CDC multi-table 
writers, including propagation from database synchronization table options.
   
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to