I don't have a solution, but I thought I'd chime in with interest in finding a solution to this problem. We have a use case where we're partitioning the dataset we write to according to Kafka partitions and having to close all writers and re-open after a rebalance is a pain point.
-Joey > On Oct 28, 2015, at 13:36, Kevin Scaldeferri <ke...@scaldeferri.com> wrote: > > One of the big issues we run into with the 0.8 high-level consumer is the > instability of partition assignments during a rebalance. The simple > lexicographic assignment strategy means that if one consumer instance dies > (and potentially a new instance with a different consumerId gets spun up to > replace it) that every other consumer instance potentially changes its > partition assignments. This is bad for data locality, cache-hit-rates, etc. > > We'd really like it if under a rebalance consumers kept the same partition > assignments as much as possible. So, if one instance goes away, another > will have to take over that partition, but there's no reason that large > numbers of consumers in the group should have to change what partition they > consumer. Similarly, if a new instance is added, other consumers might > relinquish excess partitions to maintain good balance of work-load across > the group, but it would be nice if we didn't shuffle everything around. > > I've been working on an algorithm and implementation to support this, but > before getting too far down the path, I wanted to check and see if anyone > has previously developed a consumer that works like this. So far I haven't > had any luck searching for one. > > Also, in reading the 0.9 consumer docs, there are some hints that > rebalancing / partition assignment will work differently from the current > assignment strategy (and, in particular, that it might be possible to plug > in user-defined strategies). I'd love to know a little more about this, > and if it might be simpler to achieve the behavior we're looking for with > 0.9. > > Thanks, > Kevin