fengruotj opened a new issue, #11054: URL: https://github.com/apache/paimon/issues/11054
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Paimon already supports explicit CPU and task heap resources for the committer slot sharing group through `sink.committer-cpu` and `sink.committer-memory`. However, there are no corresponding options for task off-heap or managed memory. The regular sink currently configures the committer resources as follows: ```java // FlinkSink.java:294-295: configure the committer's CPU and task heap only. configureSlotSharingGroup( committed, options.get(SINK_COMMITTER_CPU), options.get(SINK_COMMITTER_MEMORY)); ``` The combined CDC multi-table sink has the same limitation. When the committer has its own explicit resource group, increasing the writer's off-heap or managed memory request does not configure these resource dimensions for the committer group. For deployments using fine-grained resource management, users should be able to describe the committer group's CPU, task heap, task off-heap, and managed memory requirements independently. For example, a committer group may require 0.25 CPU cores, 256 MiB of task heap, 64 MiB of task off-heap, and 128 MiB of managed memory. The current committer options cannot express this resource profile. ### Solution Add two optional `MemorySize` table properties, both without default values: - `sink.committer-offheap-memory`: the task off-heap resource request for the committer slot sharing group. - `sink.committer-managed-memory`: the managed memory resource request for the committer slot sharing group. Apply the configured values through `SlotSharingGroup.Builder.setTaskOffHeapMemory()` and `SlotSharingGroup.Builder.setManagedMemory()`, respectively, while retaining the existing CPU and task heap settings. Example: ```properties sink.committer-cpu = 0.25 sink.committer-memory = 256 mb sink.committer-offheap-memory = 64 mb sink.committer-managed-memory = 128 mb ``` A local Flink 1.20.4 probe confirmed the current CPU/heap-only profile using the regular committer's resource helper and the combined CDC sink. The proposed result below is the expected behavior after implementation, not a completed validation of the new options: ```text Committer group CPU Task heap Task off-heap Managed Current explicit profile 0.25 256 MiB 0 MiB 0 MiB Proposed expected profile 0.25 256 MiB 64 MiB 128 MiB ``` The proposal has the following behavior: - Allow either new option to be configured independently, or both together. Each configured value must be positive and requires `sink.committer-memory`. Preserve the existing default of one CPU core unless `sink.committer-cpu` is configured. - Support memory units such as `kb`, `mb`, and `gb`, preserving byte precision when creating Flink resource profiles. - Apply the options to both regular global committers and combined CDC multi-table committers, including propagation from database synchronization table options. - Keep committer resources independent of writer resources. Validation errors must reference the relevant `sink.committer-*` options rather than writer options. - Treat the values as resource requests for each shared slot in the committer group, rather than a total across all committer subtasks or an exclusive per-operator memory guarantee. - Do not introduce a managed-memory allocator or consumer merely by declaring a managed budget. Keep the writer allocator switch and writer managed-memory weights unchanged; actual managed-memory use remains subject to Flink's consumers and allocation rules. - Preserve existing behavior when the options are omitted, including legacy API overloads, slot sharing, chaining configuration, parallelism, and commit semantics. These options apply to operator-based commit paths. Coordinator commit runs on the JobManager without a global committer operator, so these settings do not configure its resources or introduce an additional committer vertex. ### Anything else? Resource requests follow [Flink's fine-grained resource management](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/finegrained_resource/) semantics. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
