wardlican opened a new pull request, #3922:
URL: https://github.com/apache/amoro/pull/3922
## Why are the changes needed?
This implements the logic for dynamically and evenly distributing buckets
based on the number of AMS nodes.
Close #3921 .
## Brief change log
**AmsAssignService Functionality**
1. Periodic Allocation: Performs bucket ID allocation every 30 seconds
(configurable)
2. Node Change Detection: Detects new and offline nodes
3. Incremental Allocation: Minimizes migration, preserving existing node
buckets as much as possible
4. Load Balancing: Distributes buckets evenly across all live nodes
5. Offline Node Handling: Detects offline nodes based on a timeout mechanism
and reallocates their buckets
6. Master Node Only: Only the leader node executes the allocation logic.
**ZkBucketAssignStore Features**
1. Storage Implementation: Stores bucket assignment information based on
ZooKeeper.
2. Data Storage:
- List of bucket IDs for each node (JSON format)
- Last update timestamp for each node
3. CRUD Operations:
- Save/Update Node Assignments
- Query Node Assignments
- Delete Node Assignments
- Query All Node Assignments
4. Access Control: Only the leader node can save/delete assignment
information.
5. Path Structure:
`/{cluster}/amoro/ams/bucket-assignments/{nodeKey}/assignments` and
`/{cluster}/amoro/ams/bucket-assignments/{nodeKey}/last-update-time`
two classes that work together to implement dynamic bucket ID allocation and
storage in master-slave mode.
-
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs /
not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
