humkum opened a new issue, #113: URL: https://github.com/apache/rocketmq-flink/issues/113
In current [consistent hash allocate strategy](https://github.com/apache/rocketmq-flink/blob/main/src/main/java/org/apache/flink/connector/rocketmq/source/enumerator/allocate/ConsistentHashAllocateStrategy.java#L27), we will get unbalanced assignment caused by hash collision. For example, we have a topic named TopicTest, the topic allocate at 3 brokers: broker0, broker1, broker2, 10 queues per broker, and we have 32 task managers. The allocation result using the ConsistentHashAllocateStrategy as follows: ``` {7=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=0], 8=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=1], 9=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=2], 10=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=3], 11=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=4], 12=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=5], 13=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=6], 14=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=7], 15=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=8], 16=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=9], 17=MessageQueue [topic=TopicTest, brokerName=broker1, queueId=9], 18=MessageQueue [topic=TopicTest, brokerName=broker0, queueId=9]} allocation details, format is taskIndex=allocate num: {7=1, 8=2, 9=3, 10=3, 11=3, 12=3, 13=3, 14=3, 15=3, 16=3, 17=2, 18=1} ``` We have 32 task manager, but only 12 task manager can work normally, which means we have 20 wasted task manager. We should resolve the hash collision, I'm glad for your suggestion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
