MG>Hopefully quick response
> From: dana.pow...@gmail.com
> Date: Mon, 19 Sep 2016 11:19:00 -0700
> Subject: Consul / Zookeeper [was Re: any update on this?]
> To: users@kafka.apache.org
> CC: d...@kafka.apache.org
>
> [+ dev list]
>
> I have not worked on KAFKA-1793 directly, but I believe most of the
> work so far has been in removing all zookeeper dependencies from
> clients. The two main areas for that are (1) consumer rebalancing, and
MG>Which classes/interfaces handle consumer rebalance ?
> (2) administrative commands.
MG>Which classes/interfaces handle administrative commands ?
>
> 1) Consumer rebalancing APIs were added to the broker in 0.9. The "new
> consumer" uses these apis and does not connect directly to zookeeper
> to manage group leadership and rebalancing. So my understanding is
> that this work is complete.
MG>the TODO is to document what these classes are ..who is the tech lead on 0.9
rebalancing API?
>
> 2) Admin commands are being converted to direct API calls instead of
> direct zookeeper access with KIP-4. A small part of this project was
> released in 0.10.0.0 and are open PRs for additional chunks that may
> make it into 0.10.1.0 . If you are interested in helping or getting
> involved, you can follow the KIP-4 discussions on the dev mailing
> list.
MG>URL for the KIP-4 wiki/blog ?
>
> When the client issues are completed I think the next step will be to
> refactor the broker's zookeeper access (zkUtils) into an abstract
> interface that could potentially be provided by consul or etcd. With
> an interface in place, it should be possible to write an alternate
> implementation of that interface for consul.
MG>this is how maven implements wagon provider..you can use existing
implementations of wagon interfaceMG>or of course create your own..anyone
taking the lead on this so far?
>
> Hope this helps,
MG>Thanks for taking the time to think consul/zookeeper integration through
> -Dana
MG>looks like we have work to do..I have a few cycles available starting
Sunday..so I would definitely like to "lend a hand"
> On Mon, Sep 19, 2016 at 6:31 AM, Martin Gainty wrote:
> > Jens/Kant
> > not aware of any shortfall with zookeeper so perhaps you can suggest
> > advantages for Consul vs Zookeeper?
> > Maven (I am building, testing and running kafka internally with maven)
> > implements wagon-providers for URLConnection vs HttpURLConnection
> > wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
> > Thinking a network_provider should work for integrating external network
> > provider. how would you architect this integration?
> >
> > would a configurable network-provider such as maven-wagon-provider work for
> > kafka?Martin
> >
> >> From: kanth...@gmail.com
> >> To: users@kafka.apache.org
> >> Subject: Re: any update on this?
> >> Date: Mon, 19 Sep 2016 09:41:10 +
> >>
> >> Yes ofcourse the goal shouldn't be moving towards consul. It should just be
> >> flexible enough for users to pick any distributed coordinated system.
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se
> >> wrote:
> >> I think I read somewhere that the long-term goal is to make Kafka
> >>
> >> independent of Zookeeper alltogether. Maybe not worth spending time on
> >>
> >> migrating to Consul in that case.
> >>
> >>
> >>
> >>
> >> Cheers,
> >>
> >> Jens
> >>
> >>
> >>
> >>
> >> On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain
> >>
> >> wrote:
> >>
> >>
> >>
> >>
> >> > +2 watching.
> >>
> >> >
> >>
> >> > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali wrote:
> >>
> >> >
> >>
> >> > > https://issues.apache.org/jira/browse/KAFKA-1793
> >>
> >> > > It would be great to use Consul instead of Zookeeper for Kafka and I
> >>
> >> > think
> >>
> >> > > it
> >>
> >> > > would benefit Kafka a lot from the exponentially growing consul
> >>
> >> > community.
> >>
> >> >
> >>
> >> >
> >>
> >> >
> >>
> >> >
> >>
> >> > --
> >>
> >> >
> >>
> >> >
> >>
> >> > Jennifer Fountain
> >>
> >> > DevOPS
> >>
> >> >
> >>
> >> --
> >>
> >>
> >>
> >>
> >> Jens Rantil
> >>
> >> Backend Developer @ Tink
> >>
> >>
> >>
> >>
> >> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> >>
> >> For urgent matters you can reach me at +46-708-84 18 32.
> >
