Hi Srinivas, Your approach sounds fine, as long as you don’t need the view of the assignment to be strictly consistent. As a roughy approximation, it could work.
On the other hand, if you’re writing a custom assignor, you could consider using the SubscriptionInfo field of the joinGroup request to encode arbitrary information from the members to the leader, which it can use in making decisions. So you could encode a custom “node id” there and not have to rely on patterns in the group id. Or you could even just directly encode node load information and use that to influence the assignment. Iirc, there’s no explicit “trigger rebalance” command, but you can still make it happen by doing something like unsubscribing and resubscribing again. I hope this helps! John On Thu, Jan 30, 2020, at 09:25, Devaki, Srinivas wrote: > Also, want to clarify one more doubt, > > is there any way for the client to explicitly trigger a rebalance without > dying itself? > > On Thu, Jan 30, 2020 at 7:54 PM Devaki, Srinivas <me@eightnoteight.space> > wrote: > > > Hi All, > > > > We have a set of logstash consumer groups running under the same set of > > instances, we have decided to run separate consumer groups subscribing > > multiple topics instead of running single consumer group for all topics(the > > reasoning behind this decision is because of how our elasticsearch cluster > > is designed). > > > > Since we are running multiple consumer groups, sometimes we have detected > > that a few ec2 nodes are receiving multiple high throughput topics in > > different consumer groups. which was expected based on the implementation > > of round robin assignor. > > > > So I've decided to make a partition assignor which will consider the > > assignment based on other consumer group assignment. > > > > Could you please give me some pointers on how to proceed. This is my > > initial ideas on the problem. > > > > Solution #0: > > write an assignor, and use a specific consumer id pattern across all > > consumer groups, and in the assignor do a describe on all consumer groups > > and based on the topic throughput and the other consumer group assignment > > decide the assignment of this topic > > > > > > > > > > Thanks > > >
