fhan688 opened a new pull request, #3622:
URL: https://github.com/apache/fluss/pull/3622
### Purpose
Linked issue: https://github.com/apache/fluss/issues/3348
https://github.com/apache/fluss/issues/3536
This PR is derived from the previous rebalance throttling work in #3498
and #3537.
#3537 adds `coordinator.rebalance.max-inflight-tasks` to control how many
bucket-level rebalance tasks can run concurrently. #3498 explored limiting
large rebalance plans by splitting them into rounds, but that approach
introduced new ZooKeeper metadata models, serde, and znodes.
This PR keeps the round-limiting capability from #3498, but reworks it on
top of #3537 with a lower-intrusion design: the coordinator uses an in-memory
round/window inside `RebalanceManager` and does not add any new ZooKeeper
model, serde, or znode.
The default behavior remains unchanged.
### Brief change log
- Add `coordinator.rebalance.max-buckets-per-round` with default value `0`.
- `0` disables round limiting.
- A positive value limits how many bucket-level rebalance tasks are
admitted into one in-memory scheduling round.
- Extend `RebalanceManager` with an in-memory rebalance round/window.
- Keep the full rebalance plan persisted in the existing legacy
`RebalanceTask` znode.
- Sort bucket tasks deterministically by `tableId`, `partitionId`, and
`bucketId`.
- Admit at most `coordinator.rebalance.max-buckets-per-round` bucket
tasks into the current round.
- Open the next round only after the current round has no pending or
in-flight bucket tasks.
- Continue using `coordinator.rebalance.max-inflight-tasks` from #3537
to control concurrent execution within the active round.
- Keep the implementation ZooKeeper-compatible with the existing rebalance
flow.
- No new ZooKeeper metadata model.
- No new JSON serde.
- No new znode under the rebalance path.
- Recovery still uses the existing persisted `RebalanceTask` and
rebuilds the in-memory scheduling window after CoordinatorServer restart.
- Update rebalance documentation to explain how:
- `coordinator.rebalance.max-inflight-tasks` controls concurrent bucket
movements.
- `coordinator.rebalance.max-buckets-per-round` controls per-round
admission.
- Very small round sizes, such as `1`, can create many small scheduling
rounds and should be used only when strict serial admission is required.
### Tests
- `git diff --check`
- `mvn -pl fluss-server -am -DskipITs -Dcheckstyle.skip=true
-DfailIfNoTests=false -Dtest=RebalanceManagerTest test`
### API and Format
- Adds a new coordinator configuration:
- `coordinator.rebalance.max-buckets-per-round`
- No public client API changes.
- No RPC protocol changes.
- No storage format changes.
- No new ZooKeeper metadata, serde, or znode.
Backward compatibility:
- The default value `0` preserves the existing behavior.
- Existing rebalance task metadata remains the only persisted rebalance
execution format.
- `coordinator.rebalance.max-inflight-tasks` keeps the #3537 behavior and
remains responsible for concurrent execution throttling.
### Documentation
- Add `coordinator.rebalance.max-buckets-per-round` to
`website/docs/maintenance/configuration.md`.
- Update `website/docs/maintenance/operations/rebalance.md` with
operational guidance for using `coordinator.rebalance.max-inflight-tasks` and
`coordinator.rebalance.max-buckets-
per-round` together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
