Re: Consumer rebalancing based on partition sizes?

2015-06-23 Thread Ewen Cheslack-Postava
Current partition assignment only has a few limited options -- see the partition.assignment.strategy consumer option (which seems to be listed twice, see the second version for a more detailed explanation). There has been some discussion of making assignment strategies user extensible to support

Consumer rebalancing based on partition sizes?

2015-06-23 Thread Joel Ohman
Hello! I'm working with a topic of largely variable partition sizes. My biggest concern is that I have no control over which keys are assigned to which consumers in my consumer group, as the amount of data my consumer sees is directly reflected on it's work load. Is there a way to distribute