RE: Consul / Zookeeper [was Re: any update on this?]

2016-09-24 Thread Martin Gainty 

MG>Hopefully quick response

> From: dana.pow...@gmail.com
> Date: Mon, 19 Sep 2016 11:19:00 -0700
> Subject: Consul / Zookeeper [was Re: any update on this?]
> To: users@kafka.apache.org
> CC: d...@kafka.apache.org
> 
> [+ dev list]
> 
> I have not worked on KAFKA-1793 directly, but I believe most of the
> work so far has been in removing all zookeeper dependencies from
> clients. The two main areas for that are   (1) consumer rebalancing, and
MG>Which classes/interfaces handle consumer rebalance ?
> (2) administrative commands.
MG>Which classes/interfaces handle administrative commands ?
> 
> 1) Consumer rebalancing APIs were added to the broker in 0.9. The "new
> consumer" uses these apis and does not connect directly to zookeeper
> to manage group leadership and rebalancing. So my understanding is
> that this work is complete.
MG>the TODO is to document what these classes are ..who is the tech lead on 0.9 
rebalancing API?
> 
> 2) Admin commands are being converted to direct API calls instead of
> direct zookeeper access with KIP-4. A small part of this project was
> released in 0.10.0.0 and are open PRs for additional chunks that may
> make it into 0.10.1.0 . If you are interested in helping or getting
> involved, you can follow the KIP-4 discussions on the dev mailing
> list.
MG>URL for the KIP-4 wiki/blog ?
> 
> When the client issues are completed I think the next step will be to
> refactor the broker's zookeeper access (zkUtils) into an abstract
> interface that could potentially be provided by consul or etcd. With
> an interface in place, it should be possible to write an alternate
> implementation of that interface for consul.

MG>this is how maven implements wagon provider..you can use existing 
implementations of wagon interfaceMG>or of course create your own..anyone 
taking the lead on this so far?
> 
> Hope this helps,

MG>Thanks for taking the time to  think consul/zookeeper integration through
> -Dana

MG>looks like we have work to do..I have a few cycles available starting 
Sunday..so I would definitely like to "lend a hand"
> On Mon, Sep 19, 2016 at 6:31 AM, Martin Gainty <mgai...@hotmail.com> wrote:
> > Jens/Kant
> > not aware of any shortfall with zookeeper so perhaps you can suggest 
> > advantages for Consul vs Zookeeper?
> > Maven (I am building, testing and running kafka internally with maven) 
> > implements wagon-providers for URLConnection vs HttpURLConnection 
> > wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
> > Thinking a network_provider should work for integrating external network 
> > provider. how would you architect this integration?
> >
> > would a configurable network-provider such as maven-wagon-provider work for 
> > kafka?Martin
> >
> >> From: kanth...@gmail.com
> >> To: users@kafka.apache.org
> >> Subject: Re: any update on this?
> >> Date: Mon, 19 Sep 2016 09:41:10 +
> >>
> >> Yes ofcourse the goal shouldn't be moving towards consul. It should just be
> >> flexible enough for users to pick any distributed coordinated system.
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se
> >> wrote:
> >> I think I read somewhere that the long-term goal is to make Kafka
> >>
> >> independent of Zookeeper alltogether. Maybe not worth spending time on
> >>
> >> migrating to Consul in that case.
> >>
> >>
> >>
> >>
> >> Cheers,
> >>
> >> Jens
> >>
> >>
> >>
> >>
> >> On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfount...@meetme.com>
> >>
> >> wrote:
> >>
> >>
> >>
> >>
> >> > +2 watching.
> >>
> >> >
> >>
> >> > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <kanth...@gmail.com> wrote:
> >>
> >> >
> >>
> >> > > https://issues.apache.org/jira/browse/KAFKA-1793
> >>
> >> > > It would be great to use Consul instead of Zookeeper for Kafka and I
> >>
> >> > think
> >>
> >> > > it
> >>
> >> > > would benefit Kafka a lot from the exponentially growing consul
> >>
> >> > community.
> >>
> >> >
> >>
> >> >
> >>
> >> >
> >>
> >> >
> >>
> >> > --
> >>
> >> >
> >>
> >> >
> >>
> >> > Jennifer Fountain
> >>
> >> > DevOPS
> >>
> >> >
> >>
> >> --
> >>
> >>
> >>
> >>
> >> Jens Rantil
> >>
> >> Backend Developer @ Tink
> >>
> >>
> >>
> >>
> >> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> >>
> >> For urgent matters you can reach me at +46-708-84 18 32.
> >

Re: any update on this?

2016-09-20 Thread F21 
One possible solution might be to use parkeeper, which uses consul as 
the backend and exposes a facade that looks like zookeeper: 
https://github.com/glerchundi/parkeeper


The project doesn't seem very active though and it is unclear whether it 
supports all the features that are used by kafka.


It might be worth a try though.

Francis

On 20/09/2016 6:29 PM, Jens Rantil wrote:

not aware of any shortfall with zookeeper so perhaps you can suggest

advantages for Consul vs Zookeeper?

Maybe it's somewhat off-topic here, but Consul has several advantages over
Zookeeper:
  * It's IMHO easier to maintain, add leader nodes, remove leader nodes etc.
  * Has high level service discovery mechanisms supporting both healthchecks
and and a flexible DNS hook to load balance to services.
  * Talks HTTP, so it's much easier to integrate with any programming
language/platform.
  * Support for multiple datacenters.
  * Zookeeper supports keep-alive as healthcheck. Consul supports more
high-level healthchecks.

While Consul is more opinionated, it also has more features and highlevel.
IMHO, that makes it easier to work with not have to write thick clients.
You can read more about Consul vs. Zookeeper here:
https://www.consul.io/intro/vs/zookeeper.html

Above said, just like Kant wrote, I don't think moving from Zookeeper to
Consul should be a goal. The above Consul pros are reasons for systems to
use Consul instead of Zookeeper. We would be happy if we could ditch
Zookeeper and use Consul for all coordination to avoid the overhead of
having to maintain two highly consistent datastores.

Cheers,
Jens

On Mon, Sep 19, 2016 at 3:31 PM Martin Gainty <mgai...@hotmail.com> wrote:


Jens/Kant
not aware of any shortfall with zookeeper so perhaps you can suggest
advantages for Consul vs Zookeeper?
Maven (I am building, testing and running kafka internally with maven)
implements wagon-providers for URLConnection vs HttpURLConnection
wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
Thinking a network_provider should work for integrating external network
provider. how would you architect this integration?

would a configurable network-provider such as maven-wagon-provider work
for kafka?Martin


From: kanth...@gmail.com
To: users@kafka.apache.org
Subject: Re: any update on this?
Date: Mon, 19 Sep 2016 09:41:10 +

Yes ofcourse the goal shouldn't be moving towards consul. It should just

be

flexible enough for users to pick any distributed coordinated system.






On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se
wrote:
I think I read somewhere that the long-term goal is to make Kafka

independent of Zookeeper alltogether. Maybe not worth spending time on

migrating to Consul in that case.




Cheers,

Jens




On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfount...@meetme.com


wrote:





+2 watching.
On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <kanth...@gmail.com>

wrote:

https://issues.apache.org/jira/browse/KAFKA-1793
It would be great to use Consul instead of Zookeeper for Kafka and I

think

it
would benefit Kafka a lot from the exponentially growing consul

community.
--
Jennifer Fountain
DevOPS

--




Jens Rantil

Backend Developer @ Tink




Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden

For urgent matters you can reach me at +46-708-84 18 32.

Re: any update on this?

2016-09-20 Thread Jens Rantil 
> not aware of any shortfall with zookeeper so perhaps you can suggest
advantages for Consul vs Zookeeper?

Maybe it's somewhat off-topic here, but Consul has several advantages over
Zookeeper:
 * It's IMHO easier to maintain, add leader nodes, remove leader nodes etc.
 * Has high level service discovery mechanisms supporting both healthchecks
and and a flexible DNS hook to load balance to services.
 * Talks HTTP, so it's much easier to integrate with any programming
language/platform.
 * Support for multiple datacenters.
 * Zookeeper supports keep-alive as healthcheck. Consul supports more
high-level healthchecks.

While Consul is more opinionated, it also has more features and highlevel.
IMHO, that makes it easier to work with not have to write thick clients.
You can read more about Consul vs. Zookeeper here:
https://www.consul.io/intro/vs/zookeeper.html

Above said, just like Kant wrote, I don't think moving from Zookeeper to
Consul should be a goal. The above Consul pros are reasons for systems to
use Consul instead of Zookeeper. We would be happy if we could ditch
Zookeeper and use Consul for all coordination to avoid the overhead of
having to maintain two highly consistent datastores.

Cheers,
Jens

On Mon, Sep 19, 2016 at 3:31 PM Martin Gainty <mgai...@hotmail.com> wrote:

> Jens/Kant
> not aware of any shortfall with zookeeper so perhaps you can suggest
> advantages for Consul vs Zookeeper?
> Maven (I am building, testing and running kafka internally with maven)
> implements wagon-providers for URLConnection vs HttpURLConnection
> wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
> Thinking a network_provider should work for integrating external network
> provider. how would you architect this integration?
>
> would a configurable network-provider such as maven-wagon-provider work
> for kafka?Martin
>
> > From: kanth...@gmail.com
> > To: users@kafka.apache.org
> > Subject: Re: any update on this?
> > Date: Mon, 19 Sep 2016 09:41:10 +
> >
> > Yes ofcourse the goal shouldn't be moving towards consul. It should just
> be
> > flexible enough for users to pick any distributed coordinated system.
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se
> > wrote:
> > I think I read somewhere that the long-term goal is to make Kafka
> >
> > independent of Zookeeper alltogether. Maybe not worth spending time on
> >
> > migrating to Consul in that case.
> >
> >
> >
> >
> > Cheers,
> >
> > Jens
> >
> >
> >
> >
> > On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfount...@meetme.com
> >
> >
> > wrote:
> >
> >
> >
> >
> > > +2 watching.
> >
> > >
> >
> > > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <kanth...@gmail.com>
> wrote:
> >
> > >
> >
> > > > https://issues.apache.org/jira/browse/KAFKA-1793
> >
> > > > It would be great to use Consul instead of Zookeeper for Kafka and I
> >
> > > think
> >
> > > > it
> >
> > > > would benefit Kafka a lot from the exponentially growing consul
> >
> > > community.
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > --
> >
> > >
> >
> > >
> >
> > > Jennifer Fountain
> >
> > > DevOPS
> >
> > >
> >
> > --
> >
> >
> >
> >
> > Jens Rantil
> >
> > Backend Developer @ Tink
> >
> >
> >
> >
> > Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> >
> > For urgent matters you can reach me at +46-708-84 18 32.
>

-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Consul / Zookeeper [was Re: any update on this?]

2016-09-19 Thread Dana Powers 
[+ dev list]

I have not worked on KAFKA-1793 directly, but I believe most of the
work so far has been in removing all zookeeper dependencies from
clients. The two main areas for that are (1) consumer rebalancing, and
(2) administrative commands.

1) Consumer rebalancing APIs were added to the broker in 0.9. The "new
consumer" uses these apis and does not connect directly to zookeeper
to manage group leadership and rebalancing. So my understanding is
that this work is complete.

2) Admin commands are being converted to direct API calls instead of
direct zookeeper access with KIP-4. A small part of this project was
released in 0.10.0.0 and are open PRs for additional chunks that may
make it into 0.10.1.0 . If you are interested in helping or getting
involved, you can follow the KIP-4 discussions on the dev mailing
list.

When the client issues are completed I think the next step will be to
refactor the broker's zookeeper access (zkUtils) into an abstract
interface that could potentially be provided by consul or etcd. With
an interface in place, it should be possible to write an alternate
implementation of that interface for consul.

Hope this helps,

-Dana

On Mon, Sep 19, 2016 at 6:31 AM, Martin Gainty <mgai...@hotmail.com> wrote:
> Jens/Kant
> not aware of any shortfall with zookeeper so perhaps you can suggest 
> advantages for Consul vs Zookeeper?
> Maven (I am building, testing and running kafka internally with maven) 
> implements wagon-providers for URLConnection vs HttpURLConnection 
> wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
> Thinking a network_provider should work for integrating external network 
> provider. how would you architect this integration?
>
> would a configurable network-provider such as maven-wagon-provider work for 
> kafka?Martin
>
>> From: kanth...@gmail.com
>> To: users@kafka.apache.org
>> Subject: Re: any update on this?
>> Date: Mon, 19 Sep 2016 09:41:10 +
>>
>> Yes ofcourse the goal shouldn't be moving towards consul. It should just be
>> flexible enough for users to pick any distributed coordinated system.
>>
>>
>>
>>
>>
>>
>> On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se
>> wrote:
>> I think I read somewhere that the long-term goal is to make Kafka
>>
>> independent of Zookeeper alltogether. Maybe not worth spending time on
>>
>> migrating to Consul in that case.
>>
>>
>>
>>
>> Cheers,
>>
>> Jens
>>
>>
>>
>>
>> On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfount...@meetme.com>
>>
>> wrote:
>>
>>
>>
>>
>> > +2 watching.
>>
>> >
>>
>> > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <kanth...@gmail.com> wrote:
>>
>> >
>>
>> > > https://issues.apache.org/jira/browse/KAFKA-1793
>>
>> > > It would be great to use Consul instead of Zookeeper for Kafka and I
>>
>> > think
>>
>> > > it
>>
>> > > would benefit Kafka a lot from the exponentially growing consul
>>
>> > community.
>>
>> >
>>
>> >
>>
>> >
>>
>> >
>>
>> > --
>>
>> >
>>
>> >
>>
>> > Jennifer Fountain
>>
>> > DevOPS
>>
>> >
>>
>> --
>>
>>
>>
>>
>> Jens Rantil
>>
>> Backend Developer @ Tink
>>
>>
>>
>>
>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>>
>> For urgent matters you can reach me at +46-708-84 18 32.
>

RE: any update on this?

2016-09-19 Thread Martin Gainty 
Jens/Kant
not aware of any shortfall with zookeeper so perhaps you can suggest advantages 
for Consul vs Zookeeper?
Maven (I am building, testing and running kafka internally with maven) 
implements wagon-providers for URLConnection vs HttpURLConnection 
wagonshttps://maven.apache.org/guides/mini/guide-wagon-providers.html
Thinking a network_provider should work for integrating external network 
provider. how would you architect this integration?

would a configurable network-provider such as maven-wagon-provider work for 
kafka?Martin

> From: kanth...@gmail.com
> To: users@kafka.apache.org
> Subject: Re: any update on this?
> Date: Mon, 19 Sep 2016 09:41:10 +
> 
> Yes ofcourse the goal shouldn't be moving towards consul. It should just be
> flexible enough for users to pick any distributed coordinated system.
>  
> 
> 
> 
> 
> 
> On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se
> wrote:
> I think I read somewhere that the long-term goal is to make Kafka
> 
> independent of Zookeeper alltogether. Maybe not worth spending time on
> 
> migrating to Consul in that case.
> 
> 
> 
> 
> Cheers,
> 
> Jens
> 
> 
> 
> 
> On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain <jfount...@meetme.com>
> 
> wrote:
> 
> 
> 
> 
> > +2 watching.
> 
> >
> 
> > On Sat, Sep 17, 2016 at 2:45 AM, kant kodali <kanth...@gmail.com> wrote:
> 
> >
> 
> > > https://issues.apache.org/jira/browse/KAFKA-1793
> 
> > > It would be great to use Consul instead of Zookeeper for Kafka and I
> 
> > think
> 
> > > it
> 
> > > would benefit Kafka a lot from the exponentially growing consul
> 
> > community.
> 
> >
> 
> >
> 
> >
> 
> >
> 
> > --
> 
> >
> 
> >
> 
> > Jennifer Fountain
> 
> > DevOPS
> 
> >
> 
> -- 
> 
> 
> 
> 
> Jens Rantil
> 
> Backend Developer @ Tink
> 
> 
> 
> 
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> 
> For urgent matters you can reach me at +46-708-84 18 32.

Re: any update on this?

2016-09-19 Thread kant kodali 
Yes ofcourse the goal shouldn't be moving towards consul. It should just be
flexible enough for users to pick any distributed coordinated system.
 





On Mon, Sep 19, 2016 2:23 AM, Jens Rantil jens.ran...@tink.se
wrote:
I think I read somewhere that the long-term goal is to make Kafka

independent of Zookeeper alltogether. Maybe not worth spending time on

migrating to Consul in that case.




Cheers,

Jens




On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain 

wrote:




> +2 watching.

>

> On Sat, Sep 17, 2016 at 2:45 AM, kant kodali  wrote:

>

> > https://issues.apache.org/jira/browse/KAFKA-1793

> > It would be great to use Consul instead of Zookeeper for Kafka and I

> think

> > it

> > would benefit Kafka a lot from the exponentially growing consul

> community.

>

>

>

>

> --

>

>

> Jennifer Fountain

> DevOPS

>

-- 




Jens Rantil

Backend Developer @ Tink




Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden

For urgent matters you can reach me at +46-708-84 18 32.

Re: any update on this?

2016-09-19 Thread Jens Rantil 
I think I read somewhere that the long-term goal is to make Kafka
independent of Zookeeper alltogether. Maybe not worth spending time on
migrating to Consul in that case.

Cheers,
Jens

On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain 
wrote:

> +2 watching.
>
> On Sat, Sep 17, 2016 at 2:45 AM, kant kodali  wrote:
>
> > https://issues.apache.org/jira/browse/KAFKA-1793
> > It would be great to use Consul instead of Zookeeper for Kafka and I
> think
> > it
> > would benefit Kafka a lot from the exponentially growing consul
> community.
>
>
>
>
> --
>
>
> Jennifer Fountain
> DevOPS
>
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: any update on this?

2016-09-17 Thread Jennifer Fountain 
+2 watching.

On Sat, Sep 17, 2016 at 2:45 AM, kant kodali  wrote:

> https://issues.apache.org/jira/browse/KAFKA-1793
> It would be great to use Consul instead of Zookeeper for Kafka and I think
> it
> would benefit Kafka a lot from the exponentially growing consul community.




-- 


Jennifer Fountain
DevOPS