lifulong opened a new pull request, #4508:
URL: https://github.com/apache/flink-cdc/pull/4508
…skew
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## What is the purpose of this pull request?
TiDB CDC `RichParallelSourceFunction` used to split a table by evenly
partitioning the entire `long` handle space (`Long.MIN_VALUE` ~
`Long.MAX_VALUE`).
For clustered `AUTO_INCREMENT` primary keys, real handles only occupy a tiny
slice of that space (e.g. `id ∈ [5.6e6, 2.6e7]`). With parallelism 8, almost
all rows land on a single subtask (typically `(5/8)`), while the others finish
snapshot in milliseconds. That subtask then OOMs during `KVClient.scan`,
checkpoints never complete, and the Paimon sink commits nothing.
This change assigns each subtask a contiguous chunk of **real TiKV regions**
covering the table, so snapshot and incremental CDC scale with region count
instead of the theoretical `long` domain.
## Brief change log
- Query PD via `RangeSplitter.splitRangeByRegion` and assign contiguous
region chunks to each parallel subtask
- Clip region ranges to the table key range, drop empties, and **sort by
start key** before merging (splitter returns HashMap order, not key order)
- On region-meta failure, assign the full table to subtask 0 only — do
**not** fall back to handle-space split
- Keep the old handle-space helpers as deprecated fallback APIs
- Add unit tests for balanced chunks, unsorted input, table-range clipping,
and parallelism > region count
## Verifying this change
- *Added/Updated unit tests in `TableKeyRangeUtilsTest`*
- *Manually tested by our produce env job*
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs /
not documented)
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change the checkbox below to `[X]` followed by the name of the tool, and
uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this
PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed.
-->
- [ ] Yes (cursor)
<!--
Generated-by: [Tool Name and Version]
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
