fengruotj opened a new issue, #11054:
URL: https://github.com/apache/paimon/issues/11054

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Motivation
   
     Paimon already supports explicit CPU and task heap resources for the 
committer slot sharing group through `sink.committer-cpu` and 
`sink.committer-memory`. However, there are no corresponding options for task 
off-heap or managed memory. 
    
     The regular sink currently configures the committer resources as follows: 
    
     ```java 
     // FlinkSink.java:294-295: configure the committer's CPU and task heap 
only. 
     configureSlotSharingGroup( 
             committed, options.get(SINK_COMMITTER_CPU), 
options.get(SINK_COMMITTER_MEMORY)); 
     ``` 
    
     The combined CDC multi-table sink has the same limitation. When the 
committer has its own explicit resource group, increasing the writer's off-heap 
or managed memory request does not configure these resource dimensions for the 
committer group. 
    
     For deployments using fine-grained resource management, users should be 
able to describe the committer group's CPU, task heap, task off-heap, and 
managed memory requirements independently. For example, a committer group may 
require 0.25 CPU cores, 256 MiB of
     task heap, 64 MiB of task off-heap, and 128 MiB of managed memory. The 
current committer options cannot express this resource profile.
   
   ### Solution
   
     Add two optional `MemorySize` table properties, both without default 
values: 
    
     - `sink.committer-offheap-memory`: the task off-heap resource request for 
the committer slot sharing group. 
     - `sink.committer-managed-memory`: the managed memory resource request for 
the committer slot sharing group. 
    
     Apply the configured values through 
`SlotSharingGroup.Builder.setTaskOffHeapMemory()` and 
`SlotSharingGroup.Builder.setManagedMemory()`, respectively, while retaining 
the existing CPU and task heap settings. 
    
     Example: 
    
     ```properties 
     sink.committer-cpu = 0.25 
     sink.committer-memory = 256 mb 
     sink.committer-offheap-memory = 64 mb 
     sink.committer-managed-memory = 128 mb 
     ``` 
    
     A local Flink 1.20.4 probe confirmed the current CPU/heap-only profile 
using the regular committer's resource helper and the combined CDC sink. The 
proposed result below is the expected behavior after implementation, not a 
completed validation of the new
     options: 
    
     ```text 
     Committer group             CPU     Task heap    Task off-heap    Managed 
     Current explicit profile   0.25    256 MiB        0 MiB           0 MiB 
     Proposed expected profile  0.25    256 MiB       64 MiB         128 MiB 
     ``` 
    
     The proposal has the following behavior: 
    
     - Allow either new option to be configured independently, or both 
together. Each configured value must be positive and requires 
`sink.committer-memory`. Preserve the existing default of one CPU core unless 
`sink.committer-cpu` is configured. 
     - Support memory units such as `kb`, `mb`, and `gb`, preserving byte 
precision when creating Flink resource profiles. 
     - Apply the options to both regular global committers and combined CDC 
multi-table committers, including propagation from database synchronization 
table options. 
     - Keep committer resources independent of writer resources. Validation 
errors must reference the relevant `sink.committer-*` options rather than 
writer options. 
     - Treat the values as resource requests for each shared slot in the 
committer group, rather than a total across all committer subtasks or an 
exclusive per-operator memory guarantee. 
     - Do not introduce a managed-memory allocator or consumer merely by 
declaring a managed budget. Keep the writer allocator switch and writer 
managed-memory weights unchanged; actual managed-memory use remains subject to 
Flink's consumers and allocation rules. 
     - Preserve existing behavior when the options are omitted, including 
legacy API overloads, slot sharing, chaining configuration, parallelism, and 
commit semantics. 
    
     These options apply to operator-based commit paths. Coordinator commit 
runs on the JobManager without a global committer operator, so these settings 
do not configure its resources or introduce an additional committer vertex.
   
   ### Anything else?
   
   Resource requests follow [Flink's fine-grained resource 
management](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/finegrained_resource/)
 semantics.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to