fengruotj opened a new pull request, #11069:
URL: https://github.com/apache/paimon/pull/11069
### Purpose
Closes #11054.
Introduce `sink.committer-managed-memory` and
`sink.committer-offheap-memory` for regular global committers and combined CDC
multi-table committers, including propagation from database synchronization
table options. Both options have no default value, must be positive, and
require `sink.committer-memory`. The existing CPU default of one core still
applies unless `sink.committer-cpu` is set.
For example, the following configuration requests 256 MiB of task heap, 64
MiB of task off-heap, and 128 MiB of managed memory for each shared slot in the
committer group:
```properties
sink.committer-memory = 256 mb
sink.committer-offheap-memory = 64 mb
sink.committer-managed-memory = 128 mb
```
Committer budgets are independent of explicit writer budgets and do not
change the writer's managed-memory allocator or allocation weight. These
settings do not apply to the JobManager-side coordinator commit path.
### Tests
- `CommitterMemoryTest`: StreamGraph and JobGraph resource propagation,
byte precision, writer/committer resource independence with the managed
allocator enabled and disabled, missing heap and invalid values, and unchanged
defaults.
- `FlinkCdcMultiTableSinkTest`: database-builder propagation of committer
budgets, writer/committer resource separation, configuration validation, and
legacy-constructor compatibility.
- `WriterOffHeapMemoryTest` and `WriterManagedMemoryTest`: regression
coverage for existing writer resource requests and managed-memory allocation
behavior.
- `FlinkSinkTest`: regression coverage for coordinator commit
prerequisites and checkpoint compatibility.
Local validation passed with zero failures, errors, or skipped tests:
- Flink 1 profile: 68 tests across the five classes above (common: Flink
1.20.1; CDC: Flink 1.20.4; JDK 11).
- Flink 2 profile: 54 tests across the four common-module classes (Flink
2.2.0; JDK 17).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]