fengruotj opened a new issue, #11052: URL: https://github.com/apache/paimon/issues/11052
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Paimon exposes `sink.writer-cpu`, `sink.writer-memory`, and `sink.writer-managed-memory` to configure the writer slot sharing group's CPU, task heap, and managed memory resources. There is currently no corresponding option for task off-heap memory. When the writer uses an explicit resource profile, its task off-heap request remains zero. Workloads that need native or direct memory on the write path therefore cannot express that requirement through Paimon's writer resource options. Increasing task heap or managed memory does not configure the task off-heap component. For example, a workload may need 512 MiB of task heap, 128 MiB of task off-heap memory, and 64 MiB of managed memory per writer shared slot. Exposing the missing resource dimension would let users declare these requirements independently in deployments that support Flink's fine-grained resource management. ### Solution _No response_ ### Anything else? Introduce `sink.writer-offheap-memory` as an optional `MemorySize` table property without a default. Pass its byte value to `SlotSharingGroup.Builder.setTaskOffHeapMemory()` when constructing the writer group. Example: ```properties sink.writer-memory = 512 mb sink.writer-offheap-memory = 128 mb sink.writer-managed-memory = 64 mb ``` A local Flink 1.20.1 probe exercised the sink's shared resource-configuration helper and checked both StreamGraph and JobGraph profiles: ```text CPU Task heap Task off-heap Managed Option absent 1 512 MiB 0 MiB 64 MiB sink.writer-offheap-memory = 128 mb 1 512 MiB 128 MiB 64 MiB ``` The option describes the resource requirement of each shared slot in the writer group. It does not reserve a total amount across all writer subtasks, guarantee exclusive memory for a writer, or enforce a per-writer native-memory limit. Actual allocation depends on Flink's scheduler and available resources. The proposal has the following behavior: - Require `sink.writer-memory` and a positive off-heap value. Continue using the existing default of one CPU core unless `sink.writer-cpu` is configured. - Accept memory units such as `kb`, `mb`, and `gb`, preserving byte precision. For example, `1536 kb` produces a request of 1,572,864 bytes. - Keep task off-heap and managed memory independent. The option does not change writer-buffer allocation, enable `sink.use-managed-memory-allocator`, or alter `sink.managed.writer-buffer-memory` weights. - Support both regular Flink sink writers and combined CDC multi-table writers, including propagation from database synchronization table options. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
