JNSimba opened a new pull request, #68432:
URL: https://github.com/apache/doris/pull/68432
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
FROM TO CDC snapshot tasks used a fixed snapshot parallelism without
considering the CPU capacity of the selected backend. This could create
excessive snapshot readers and CDC client threads on smaller backends.
This change derives the effective snapshot parallelism from half of the
selected backend CPU cores, caps it with the mutable FE configuration
`streaming_cdc_max_snapshot_parallelism` (default 16), and preserves a lower
user-configured value. MySQL and OceanBase `server_id` ranges continue to cap
the effective parallelism when the user does not explicitly configure snapshot
parallelism. The offset provider sends at most that many pending splits, and
the CDC client creates one worker per split in the actual batch.
### Release note
Adapt FROM TO CDC snapshot parallelism to the selected backend CPU capacity,
with a configurable maximum of 16.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [x] Other reason: Unit tests were added, but tests were not run
for this change.
- Behavior changed:
- [ ] No.
- [x] Yes. Snapshot parallelism is automatically capped by the selected
backend CPU capacity and the FE maximum.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
