jack870131 opened a new issue #2149: URL: https://github.com/apache/rocketmq/issues/2149
## Problem Description The current rebalancing mechanism for the cluster mode of RocketMQ is shown below: Each consumer instance starts a rebalance service thread, which periodically obtains a list of consumer instances under the group from the broker, and sorts the message queue under the topic and consumer instances under the group, and then uses related strategy to calculate the message queue to be pulled. To be specific, RocketMQ guarantees data consistency through a full link method. For example, each broker regularly registers with each name server to ensure that the routing information between name servers are the same. The heartbeat combined with the subscription information is sent to each broker through the consumer instance to ensure that the subscription information between brokers are the same. However, such architecture design will bring inconsistent views in scenarios such as unreliable networks and delays. In terms of load balancing, consumers need to obtain routing information (topic-queue list) and subscription information (cid list). Each consumer instance will obtain the routing information from a random name server, and the subscription information from a random broker. The inconsistency of the view between each consumer instance will lead to unbalanced load. Except for the problem of the load balancing architecture, there is also another problem of queue allocating strategies implemented in RocketMQ, which existing algorithms lack of stickiness so that previous assignments of queues would not be considered. While the number of queues increases or decreases, this would lead to a large amount of overhead in the process of reassigning queues to consumers. ## Implementation 1. Move the rebalancing calculation to the broker, and the client requests the broker to obtain the allocation result. 2. Provide a sticky rebalancing algorithm for Apache RocketMQ. 3. Implementation of rebalances evaluation method in Openmessaging-Chaos. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
