Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[lbarikyn]

On 2018/09/14 21:00:10, Colin McCabe  wrote: 
> On Fri, Sep 14, 2018, at 00:46, lbari...@gmail.com wrote:
> > Hello, Colin!
> > 
> > I have read the discussion on KIP-273 and want to raise this question. 
> > Deploying Kafka to Kubernetes is a crucial problem in my current work. 
> > And usage of ZooKeeper makes troubles, because it works unstable in 
> > container. 
> 
> Hi Ibarikyn,
> 
> I haven't heard any reports of ZooKeeper being unstable in containers.  There 
> are a lot of companies already deploying Kafka and ZooKeeper this way, so I 
> am surprised to hear that you had trouble.  Can you be a little more clear 
> about what issues you are seeing with ZooKeeper in containers?
> 
> > In this discussion you have mentioned that the goal is to get rid of it 
> > in future releases. But since then there were no posts about it. Also, i 
> > can not see such plan in a roadmap. Cuold you please tell us more about 
> > work made in this stream.
> 
> ZooKeeper (and etcd, and consul, and the others) were originally designed as 
> configuration management systems.  They're really not set up to do what Kafka 
> does with them, which is use them as one half of a data management system.  
> We need things like uniform access control, the ability to cache metadata 
> locally, the ability to propagate only what has changed to brokers, etc.  
> That requires us to manage our own metadata log.  Kafka is a system for 
> managing logs, so this is a natural design.
> 
> There have been a lot of discussions about pluggable consensus before on the 
> mailing list.  Even if we wanted to keep using a configuration management 
> system for metadata, pluggable consensus is a really bad idea.  It means we 
> would have to support multiple systems, getting correspondingly less testing 
> in each.  We also would have to use least common denominator APIs, since each 
> system is somewhat different.  And configuration would get harder for new 
> users.
> 
> best,
> Colin
> 
Hello,

Thank you for the answer!
I can see your point and agree with some ideas (eg complexity of configuration 
custom systems for users).

Some troubles using Zk that I mentioned:
 - it doesnt suuport ssl (at least the version that is used in Kafka)
 - cluster being unstable
 - dns names confusing

But I talking about another issue. As far as I am concerned, getting rid of 
dependency to Zk(and other configuration management) is the right direction. 
But you did not evaluate the time when we will see it in Kafka. I would like to 
know, is it WIP or smth like that. Do u have any evaluations when this feature 
will be released?
BTW, how Zk-less version is supposed to work? What technology will be used as a 
metadata container? 
As far as I am very interested in this feature, I d like to try to contribute 
my work. Could u show me issues/tickets so i can get acquainted with the 
situation and maybe start work.

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Colin McCabe]

On Fri, Sep 14, 2018, at 00:46, lbari...@gmail.com wrote:
> Hello, Colin!
> 
> I have read the discussion on KIP-273 and want to raise this question. 
> Deploying Kafka to Kubernetes is a crucial problem in my current work. 
> And usage of ZooKeeper makes troubles, because it works unstable in 
> container. 

Hi Ibarikyn,

I haven't heard any reports of ZooKeeper being unstable in containers.  There 
are a lot of companies already deploying Kafka and ZooKeeper this way, so I am 
surprised to hear that you had trouble.  Can you be a little more clear about 
what issues you are seeing with ZooKeeper in containers?

> In this discussion you have mentioned that the goal is to get rid of it 
> in future releases. But since then there were no posts about it. Also, i 
> can not see such plan in a roadmap. Cuold you please tell us more about 
> work made in this stream.

ZooKeeper (and etcd, and consul, and the others) were originally designed as 
configuration management systems.  They're really not set up to do what Kafka 
does with them, which is use them as one half of a data management system.  We 
need things like uniform access control, the ability to cache metadata locally, 
the ability to propagate only what has changed to brokers, etc.  That requires 
us to manage our own metadata log.  Kafka is a system for managing logs, so 
this is a natural design.

There have been a lot of discussions about pluggable consensus before on the 
mailing list.  Even if we wanted to keep using a configuration management 
system for metadata, pluggable consensus is a really bad idea.  It means we 
would have to support multiple systems, getting correspondingly less testing in 
each.  We also would have to use least common denominator APIs, since each 
system is somewhat different.  And configuration would get harder for new users.

best,
Colin

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[lbarikyn]

On 2018/05/09 14:29:17, Molnár Bálint  wrote: 
> Hi,
> I just rebased the Etcd implementation proposal on trunk. Pinging to see if
> anyone has feedback on my questions from my previous email.
> 
> Molnár Bálint  ezt írta (időpont: 2018. ápr. 4.,
> Sze, 10:08):
> 
> > Hi,
> > Thanks again for the feedback.
> >
> > Is there already ongoing work for having an own consensus implementation
> > within Kafka?
> > If that work haven't started yet, we think there is value in having an
> > interim solution, that allows the use of another consensus system besides
> > Zookeeper.
> >
> > We ask the community to take a look at the Etcd implementation proposal
> > we created and provide feedback on that.
> > This helps to asses rather this approach is viable at all.
> >
> > We are open to collaborate on integrating our proposed Etcd implementation
> > into any integration test system, to certify that all use cases works as
> > expected.
> >
> > Balint
> >
> > 2018-03-30 22:21 GMT+02:00 Gwen Shapira :
> >
> >> Hi,
> >>
> >> I had an offline discussion with Ismael and wanted to summarize the
> >> comments and questions he raised so we are all on the same page.
> >>
> >> The core issue is that this change adds a new public API. Since we already
> >> know that the goal for the next 1-2 years is to get rid of ZK completely.
> >> Do we want to go to the effort of adding (and discussing and reviewing) a
> >> new public API, knowing that it will be completely removed in a year? And
> >> since building and testing a plugin also involves effort, will anyone do
> >> it
> >> for something that is going to be temporary by design?
> >>
> >> Ismael, correct me if this isn't a fair representation of your concerns.
> >>
> >> Gwen
> >>
> >>
> >>
> >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira  wrote:
> >>
> >> > Few other concerns that were raised in the previous discussion were
> >> around
> >> > the challenges both to maintainers and users in making this API
> >> pluggable
> >> > and how does making the interface pluggable aligns with future goals for
> >> > the project. At the time this was difficult to discuss because there
> >> wasn't
> >> > a concrete proposal. I want to discuss these points in the context of
> >> this
> >> > specific proposal:
> >> >
> >> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> >> > implementations to cover.
> >> > In this case: At the time, the Kafka project didn't have much
> >> > experience with pluggable APIs and components, so the concerns were very
> >> > valid. Right now Kafka has multiple pluggable components - Connectors,
> >> > converters, transformations, authentication protocols, authorization
> >> > database, coordination protocol, serializers, etc. I think that as a
> >> > community we gotten better at testing the interface, testing the very
> >> few
> >> > implementations that are included in Apache Kafka itself and allowing
> >> the
> >> > community to innovate and validate outside of the Kafka project. I don't
> >> > recall major issues either from lack of testing or from usability
> >> > perspective.
> >> >
> >> > 2. Problem: Users don't want to choose a consensus implementation, they
> >> > just don't want ZK.
> >> > In this case: I agree that users don't actually want to spend time
> >> > choosing consensus implementation and a simpler deployment model would
> >> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> >> > implementation, 99% of the users will choose to use that (a vast
> >> majority
> >> > uses our less-than-amazing authorization plugin), and the few that
> >> really
> >> > need something else for whatever reason, will be able to get what they
> >> > need. As Jake said, we need to face the fact that development
> >> trajectory of
> >> > ZK isn't amazing at the moment, that it is lacking features our users
> >> need
> >> > (SSL) and it will be good to allow the user community to explore
> >> > alternatives.
> >> >
> >> > 3. Problem: Why got to the effort of refactoring if we know we want to
> >> get
> >> > rid of ZK.
> >> > In this case: This change isn't huge, it doesn't rewrite large
> >> > portions of Kafka and it does not make the future direction any more
> >> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> >> > solution, applying it on Kafka with this KIP isn't any more challenging
> >> > than applying it to Kafka without this KIP. It is a step in an
> >> orthogonal
> >> > direction, but not opposite direction. I think that letting the perfect
> >> > become the enemy of the good is a repeated failure mode in this
> >> community.
> >> > Can we discuss whether this proposal is good even if there is a more
> >> > complex proposal that may be better? As long as they don't conflict?
> >> >
> >> > Gwen
> >> >
> >> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint 
> >> > wrote:
> >> >
> >> >> Thanks, for the feedback.
> >> >>
> >> >> Developing an internal consensus service inside 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[lbarikyn]

On 2018/05/18 13:00:52, Molnár Bálint  wrote: 
> Hi Jakub,
> 
> Yes I tried it and as I saw in various forums it works well for Kafka as
> well, but because it mimics the Zookeeper API it creates quite many
> metadata related value.
> Kafka uses only a subset from the Zookeeper API, so those values can be
> omitted by implementing Kafka Zookeeper calls in a more Etcd way.
> Also in the future I hope we can replace the Zookeeper specific interface
> inside Kafka to a more generic one.
> 
> Regards
> Balint
> 
> Jakub Scholz  ezt írta (időpont: 2018. máj. 17., Cs,
> 22:19):
> 
> > Hi Balint,
> >
> > Out of curiosity - have you ever tried this project
> > https://github.com/coreos/zetcd which tries to provide Zookeeper API for
> > Etcd?
> >
> > Thanks & Regards
> > Jakub
> >
> > On Wed, Mar 28, 2018 at 6:18 PM Molnár Bálint 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> > >
> > > Here is the link to the KIP:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> > >
> > > Looking forward to the discussion.
> > >
> > > Thanks,
> > > Balint
> > >
> >
> 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[lbarikyn]

On 2018/05/18 13:00:52, Molnár Bálint  wrote: 
> Hi Jakub,
> 
> Yes I tried it and as I saw in various forums it works well for Kafka as
> well, but because it mimics the Zookeeper API it creates quite many
> metadata related value.
> Kafka uses only a subset from the Zookeeper API, so those values can be
> omitted by implementing Kafka Zookeeper calls in a more Etcd way.
> Also in the future I hope we can replace the Zookeeper specific interface
> inside Kafka to a more generic one.
> 
> Regards
> Balint
> 
> Jakub Scholz  ezt írta (időpont: 2018. máj. 17., Cs,
> 22:19):
> 
> > Hi Balint,
> >
> > Out of curiosity - have you ever tried this project
> > https://github.com/coreos/zetcd which tries to provide Zookeeper API for
> > Etcd?
> >
> > Thanks & Regards
> > Jakub
> >
> > On Wed, Mar 28, 2018 at 6:18 PM Molnár Bálint 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> > >
> > > Here is the link to the KIP:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> > >
> > > Looking forward to the discussion.
> > >
> > > Thanks,
> > > Balint
> > >
> >
> 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi Jakub,

Yes I tried it and as I saw in various forums it works well for Kafka as
well, but because it mimics the Zookeeper API it creates quite many
metadata related value.
Kafka uses only a subset from the Zookeeper API, so those values can be
omitted by implementing Kafka Zookeeper calls in a more Etcd way.
Also in the future I hope we can replace the Zookeeper specific interface
inside Kafka to a more generic one.

Regards
Balint

Jakub Scholz  ezt írta (időpont: 2018. máj. 17., Cs,
22:19):

> Hi Balint,
>
> Out of curiosity - have you ever tried this project
> https://github.com/coreos/zetcd which tries to provide Zookeeper API for
> Etcd?
>
> Thanks & Regards
> Jakub
>
> On Wed, Mar 28, 2018 at 6:18 PM Molnár Bálint 
> wrote:
>
> > Hi all,
> >
> > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> >
> > Here is the link to the KIP:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >
> > Looking forward to the discussion.
> >
> > Thanks,
> > Balint
> >
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Jakub Scholz]

Hi Balint,

Out of curiosity - have you ever tried this project
https://github.com/coreos/zetcd which tries to provide Zookeeper API for
Etcd?

Thanks & Regards
Jakub

On Wed, Mar 28, 2018 at 6:18 PM Molnár Bálint 
wrote:

> Hi all,
>
> I have created KIP-273: Kafka to support using ETCD beside Zookeeper
>
> Here is the link to the KIP:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
>
> Looking forward to the discussion.
>
> Thanks,
> Balint
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Jakub Scholz]

Hi,

To be honest, I'm not sure I understand who is the "we" in the "We're going
to post ...". If there are any discussions or plans to replace Zookeeper,
then I think they should be done here on the developer list instead of
teasing people for several months with some secretive projects.

Thanks & Regards
Jakub


On Mon, May 14, 2018 at 9:57 AM Molnár Bálint 
wrote:

> Hi Colin,
>
> Do you have a rough estimate on that?
>
> Thanks,
> Balint
>
> Colin McCabe  ezt írta (időpont: 2018. máj. 9., Sze,
> 19:53):
>
> > Hi Molnar,
> >
> > The points Ismael brought up earlier (and that were brought up on KIP-30)
> > are still relevant here.  As Ismael said, the goal is to get rid of
> > external dependencies here.   We're going to post more about this soon
> > (sorry for the delay)
> >
> > thanks,
> > Colin
> >
> >
> > On Wed, May 9, 2018, at 07:29, Molnár Bálint wrote:
> > > Hi,
> > > I just rebased the Etcd implementation proposal on trunk. Pinging to
> see
> > if
> > > anyone has feedback on my questions from my previous email.
> > >
> > > Molnár Bálint  ezt írta (időpont: 2018. ápr.
> 4.,
> > > Sze, 10:08):
> > >
> > > > Hi,
> > > > Thanks again for the feedback.
> > > >
> > > > Is there already ongoing work for having an own consensus
> > implementation
> > > > within Kafka?
> > > > If that work haven't started yet, we think there is value in having
> an
> > > > interim solution, that allows the use of another consensus system
> > besides
> > > > Zookeeper.
> > > >
> > > > We ask the community to take a look at the Etcd implementation
> proposal
> > > > we created and provide feedback on that.
> > > > This helps to asses rather this approach is viable at all.
> > > >
> > > > We are open to collaborate on integrating our proposed Etcd
> > implementation
> > > > into any integration test system, to certify that all use cases works
> > as
> > > > expected.
> > > >
> > > > Balint
> > > >
> > > > 2018-03-30 22:21 GMT+02:00 Gwen Shapira :
> > > >
> > > >> Hi,
> > > >>
> > > >> I had an offline discussion with Ismael and wanted to summarize the
> > > >> comments and questions he raised so we are all on the same page.
> > > >>
> > > >> The core issue is that this change adds a new public API. Since we
> > already
> > > >> know that the goal for the next 1-2 years is to get rid of ZK
> > completely.
> > > >> Do we want to go to the effort of adding (and discussing and
> > reviewing) a
> > > >> new public API, knowing that it will be completely removed in a
> year?
> > And
> > > >> since building and testing a plugin also involves effort, will
> anyone
> > do
> > > >> it
> > > >> for something that is going to be temporary by design?
> > > >>
> > > >> Ismael, correct me if this isn't a fair representation of your
> > concerns.
> > > >>
> > > >> Gwen
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira 
> > wrote:
> > > >>
> > > >> > Few other concerns that were raised in the previous discussion
> were
> > > >> around
> > > >> > the challenges both to maintainers and users in making this API
> > > >> pluggable
> > > >> > and how does making the interface pluggable aligns with future
> > goals for
> > > >> > the project. At the time this was difficult to discuss because
> there
> > > >> wasn't
> > > >> > a concrete proposal. I want to discuss these points in the context
> > of
> > > >> this
> > > >> > specific proposal:
> > > >> >
> > > >> > 1. Problem: Pluggable APIs mean larger surface testing area and
> > multiple
> > > >> > implementations to cover.
> > > >> > In this case: At the time, the Kafka project didn't have much
> > > >> > experience with pluggable APIs and components, so the concerns
> were
> > very
> > > >> > valid. Right now Kafka has multiple pluggable components -
> > Connectors,
> > > >> > converters, transformations, authentication protocols,
> authorization
> > > >> > database, coordination protocol, serializers, etc. I think that
> as a
> > > >> > community we gotten better at testing the interface, testing the
> > very
> > > >> few
> > > >> > implementations that are included in Apache Kafka itself and
> > allowing
> > > >> the
> > > >> > community to innovate and validate outside of the Kafka project. I
> > don't
> > > >> > recall major issues either from lack of testing or from usability
> > > >> > perspective.
> > > >> >
> > > >> > 2. Problem: Users don't want to choose a consensus implementation,
> > they
> > > >> > just don't want ZK.
> > > >> > In this case: I agree that users don't actually want to spend
> > time
> > > >> > choosing consensus implementation and a simpler deployment model
> > would
> > > >> > serve them better. IMO, if Apache Kafka ships with our well-tested
> > ZK
> > > >> > implementation, 99% of the users will choose to use that (a vast
> > > >> majority
> > > >> > uses our less-than-amazing authorization plugin), and the few that

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi Colin,

Do you have a rough estimate on that?

Thanks,
Balint

Colin McCabe  ezt írta (időpont: 2018. máj. 9., Sze,
19:53):

> Hi Molnar,
>
> The points Ismael brought up earlier (and that were brought up on KIP-30)
> are still relevant here.  As Ismael said, the goal is to get rid of
> external dependencies here.   We're going to post more about this soon
> (sorry for the delay)
>
> thanks,
> Colin
>
>
> On Wed, May 9, 2018, at 07:29, Molnár Bálint wrote:
> > Hi,
> > I just rebased the Etcd implementation proposal on trunk. Pinging to see
> if
> > anyone has feedback on my questions from my previous email.
> >
> > Molnár Bálint  ezt írta (időpont: 2018. ápr. 4.,
> > Sze, 10:08):
> >
> > > Hi,
> > > Thanks again for the feedback.
> > >
> > > Is there already ongoing work for having an own consensus
> implementation
> > > within Kafka?
> > > If that work haven't started yet, we think there is value in having an
> > > interim solution, that allows the use of another consensus system
> besides
> > > Zookeeper.
> > >
> > > We ask the community to take a look at the Etcd implementation proposal
> > > we created and provide feedback on that.
> > > This helps to asses rather this approach is viable at all.
> > >
> > > We are open to collaborate on integrating our proposed Etcd
> implementation
> > > into any integration test system, to certify that all use cases works
> as
> > > expected.
> > >
> > > Balint
> > >
> > > 2018-03-30 22:21 GMT+02:00 Gwen Shapira :
> > >
> > >> Hi,
> > >>
> > >> I had an offline discussion with Ismael and wanted to summarize the
> > >> comments and questions he raised so we are all on the same page.
> > >>
> > >> The core issue is that this change adds a new public API. Since we
> already
> > >> know that the goal for the next 1-2 years is to get rid of ZK
> completely.
> > >> Do we want to go to the effort of adding (and discussing and
> reviewing) a
> > >> new public API, knowing that it will be completely removed in a year?
> And
> > >> since building and testing a plugin also involves effort, will anyone
> do
> > >> it
> > >> for something that is going to be temporary by design?
> > >>
> > >> Ismael, correct me if this isn't a fair representation of your
> concerns.
> > >>
> > >> Gwen
> > >>
> > >>
> > >>
> > >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira 
> wrote:
> > >>
> > >> > Few other concerns that were raised in the previous discussion were
> > >> around
> > >> > the challenges both to maintainers and users in making this API
> > >> pluggable
> > >> > and how does making the interface pluggable aligns with future
> goals for
> > >> > the project. At the time this was difficult to discuss because there
> > >> wasn't
> > >> > a concrete proposal. I want to discuss these points in the context
> of
> > >> this
> > >> > specific proposal:
> > >> >
> > >> > 1. Problem: Pluggable APIs mean larger surface testing area and
> multiple
> > >> > implementations to cover.
> > >> > In this case: At the time, the Kafka project didn't have much
> > >> > experience with pluggable APIs and components, so the concerns were
> very
> > >> > valid. Right now Kafka has multiple pluggable components -
> Connectors,
> > >> > converters, transformations, authentication protocols, authorization
> > >> > database, coordination protocol, serializers, etc. I think that as a
> > >> > community we gotten better at testing the interface, testing the
> very
> > >> few
> > >> > implementations that are included in Apache Kafka itself and
> allowing
> > >> the
> > >> > community to innovate and validate outside of the Kafka project. I
> don't
> > >> > recall major issues either from lack of testing or from usability
> > >> > perspective.
> > >> >
> > >> > 2. Problem: Users don't want to choose a consensus implementation,
> they
> > >> > just don't want ZK.
> > >> > In this case: I agree that users don't actually want to spend
> time
> > >> > choosing consensus implementation and a simpler deployment model
> would
> > >> > serve them better. IMO, if Apache Kafka ships with our well-tested
> ZK
> > >> > implementation, 99% of the users will choose to use that (a vast
> > >> majority
> > >> > uses our less-than-amazing authorization plugin), and the few that
> > >> really
> > >> > need something else for whatever reason, will be able to get what
> they
> > >> > need. As Jake said, we need to face the fact that development
> > >> trajectory of
> > >> > ZK isn't amazing at the moment, that it is lacking features our
> users
> > >> need
> > >> > (SSL) and it will be good to allow the user community to explore
> > >> > alternatives.
> > >> >
> > >> > 3. Problem: Why got to the effort of refactoring if we know we want
> to
> > >> get
> > >> > rid of ZK.
> > >> > In this case: This change isn't huge, it doesn't rewrite large
> > >> > portions of Kafka and it does not make the future direction any more
> > >> > 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Colin McCabe]

Hi Molnar,

The points Ismael brought up earlier (and that were brought up on KIP-30) are 
still relevant here.  As Ismael said, the goal is to get rid of external 
dependencies here.   We're going to post more about this soon (sorry for the 
delay)

thanks,
Colin


On Wed, May 9, 2018, at 07:29, Molnár Bálint wrote:
> Hi,
> I just rebased the Etcd implementation proposal on trunk. Pinging to see if
> anyone has feedback on my questions from my previous email.
> 
> Molnár Bálint  ezt írta (időpont: 2018. ápr. 4.,
> Sze, 10:08):
> 
> > Hi,
> > Thanks again for the feedback.
> >
> > Is there already ongoing work for having an own consensus implementation
> > within Kafka?
> > If that work haven't started yet, we think there is value in having an
> > interim solution, that allows the use of another consensus system besides
> > Zookeeper.
> >
> > We ask the community to take a look at the Etcd implementation proposal
> > we created and provide feedback on that.
> > This helps to asses rather this approach is viable at all.
> >
> > We are open to collaborate on integrating our proposed Etcd implementation
> > into any integration test system, to certify that all use cases works as
> > expected.
> >
> > Balint
> >
> > 2018-03-30 22:21 GMT+02:00 Gwen Shapira :
> >
> >> Hi,
> >>
> >> I had an offline discussion with Ismael and wanted to summarize the
> >> comments and questions he raised so we are all on the same page.
> >>
> >> The core issue is that this change adds a new public API. Since we already
> >> know that the goal for the next 1-2 years is to get rid of ZK completely.
> >> Do we want to go to the effort of adding (and discussing and reviewing) a
> >> new public API, knowing that it will be completely removed in a year? And
> >> since building and testing a plugin also involves effort, will anyone do
> >> it
> >> for something that is going to be temporary by design?
> >>
> >> Ismael, correct me if this isn't a fair representation of your concerns.
> >>
> >> Gwen
> >>
> >>
> >>
> >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira  wrote:
> >>
> >> > Few other concerns that were raised in the previous discussion were
> >> around
> >> > the challenges both to maintainers and users in making this API
> >> pluggable
> >> > and how does making the interface pluggable aligns with future goals for
> >> > the project. At the time this was difficult to discuss because there
> >> wasn't
> >> > a concrete proposal. I want to discuss these points in the context of
> >> this
> >> > specific proposal:
> >> >
> >> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> >> > implementations to cover.
> >> > In this case: At the time, the Kafka project didn't have much
> >> > experience with pluggable APIs and components, so the concerns were very
> >> > valid. Right now Kafka has multiple pluggable components - Connectors,
> >> > converters, transformations, authentication protocols, authorization
> >> > database, coordination protocol, serializers, etc. I think that as a
> >> > community we gotten better at testing the interface, testing the very
> >> few
> >> > implementations that are included in Apache Kafka itself and allowing
> >> the
> >> > community to innovate and validate outside of the Kafka project. I don't
> >> > recall major issues either from lack of testing or from usability
> >> > perspective.
> >> >
> >> > 2. Problem: Users don't want to choose a consensus implementation, they
> >> > just don't want ZK.
> >> > In this case: I agree that users don't actually want to spend time
> >> > choosing consensus implementation and a simpler deployment model would
> >> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> >> > implementation, 99% of the users will choose to use that (a vast
> >> majority
> >> > uses our less-than-amazing authorization plugin), and the few that
> >> really
> >> > need something else for whatever reason, will be able to get what they
> >> > need. As Jake said, we need to face the fact that development
> >> trajectory of
> >> > ZK isn't amazing at the moment, that it is lacking features our users
> >> need
> >> > (SSL) and it will be good to allow the user community to explore
> >> > alternatives.
> >> >
> >> > 3. Problem: Why got to the effort of refactoring if we know we want to
> >> get
> >> > rid of ZK.
> >> > In this case: This change isn't huge, it doesn't rewrite large
> >> > portions of Kafka and it does not make the future direction any more
> >> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> >> > solution, applying it on Kafka with this KIP isn't any more challenging
> >> > than applying it to Kafka without this KIP. It is a step in an
> >> orthogonal
> >> > direction, but not opposite direction. I think that letting the perfect
> >> > become the enemy of the good is a repeated failure mode in this
> >> community.
> >> > Can we discuss 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi,
I just rebased the Etcd implementation proposal on trunk. Pinging to see if
anyone has feedback on my questions from my previous email.

Molnár Bálint  ezt írta (időpont: 2018. ápr. 4.,
Sze, 10:08):

> Hi,
> Thanks again for the feedback.
>
> Is there already ongoing work for having an own consensus implementation
> within Kafka?
> If that work haven't started yet, we think there is value in having an
> interim solution, that allows the use of another consensus system besides
> Zookeeper.
>
> We ask the community to take a look at the Etcd implementation proposal
> we created and provide feedback on that.
> This helps to asses rather this approach is viable at all.
>
> We are open to collaborate on integrating our proposed Etcd implementation
> into any integration test system, to certify that all use cases works as
> expected.
>
> Balint
>
> 2018-03-30 22:21 GMT+02:00 Gwen Shapira :
>
>> Hi,
>>
>> I had an offline discussion with Ismael and wanted to summarize the
>> comments and questions he raised so we are all on the same page.
>>
>> The core issue is that this change adds a new public API. Since we already
>> know that the goal for the next 1-2 years is to get rid of ZK completely.
>> Do we want to go to the effort of adding (and discussing and reviewing) a
>> new public API, knowing that it will be completely removed in a year? And
>> since building and testing a plugin also involves effort, will anyone do
>> it
>> for something that is going to be temporary by design?
>>
>> Ismael, correct me if this isn't a fair representation of your concerns.
>>
>> Gwen
>>
>>
>>
>> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira  wrote:
>>
>> > Few other concerns that were raised in the previous discussion were
>> around
>> > the challenges both to maintainers and users in making this API
>> pluggable
>> > and how does making the interface pluggable aligns with future goals for
>> > the project. At the time this was difficult to discuss because there
>> wasn't
>> > a concrete proposal. I want to discuss these points in the context of
>> this
>> > specific proposal:
>> >
>> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
>> > implementations to cover.
>> > In this case: At the time, the Kafka project didn't have much
>> > experience with pluggable APIs and components, so the concerns were very
>> > valid. Right now Kafka has multiple pluggable components - Connectors,
>> > converters, transformations, authentication protocols, authorization
>> > database, coordination protocol, serializers, etc. I think that as a
>> > community we gotten better at testing the interface, testing the very
>> few
>> > implementations that are included in Apache Kafka itself and allowing
>> the
>> > community to innovate and validate outside of the Kafka project. I don't
>> > recall major issues either from lack of testing or from usability
>> > perspective.
>> >
>> > 2. Problem: Users don't want to choose a consensus implementation, they
>> > just don't want ZK.
>> > In this case: I agree that users don't actually want to spend time
>> > choosing consensus implementation and a simpler deployment model would
>> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
>> > implementation, 99% of the users will choose to use that (a vast
>> majority
>> > uses our less-than-amazing authorization plugin), and the few that
>> really
>> > need something else for whatever reason, will be able to get what they
>> > need. As Jake said, we need to face the fact that development
>> trajectory of
>> > ZK isn't amazing at the moment, that it is lacking features our users
>> need
>> > (SSL) and it will be good to allow the user community to explore
>> > alternatives.
>> >
>> > 3. Problem: Why got to the effort of refactoring if we know we want to
>> get
>> > rid of ZK.
>> > In this case: This change isn't huge, it doesn't rewrite large
>> > portions of Kafka and it does not make the future direction any more
>> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
>> > solution, applying it on Kafka with this KIP isn't any more challenging
>> > than applying it to Kafka without this KIP. It is a step in an
>> orthogonal
>> > direction, but not opposite direction. I think that letting the perfect
>> > become the enemy of the good is a repeated failure mode in this
>> community.
>> > Can we discuss whether this proposal is good even if there is a more
>> > complex proposal that may be better? As long as they don't conflict?
>> >
>> > Gwen
>> >
>> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint 
>> > wrote:
>> >
>> >> Thanks, for the feedback.
>> >>
>> >> Developing an internal consensus service inside Kafka would require a
>> team
>> >> dedicated to this task.
>> >> We second what Flavio said in
>> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f
>> >> 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi,
Thanks again for the feedback.

Is there already ongoing work for having an own consensus implementation
within Kafka?
If that work haven't started yet, we think there is value in having an
interim solution, that allows the use of another consensus system besides
Zookeeper.

We ask the community to take a look at the Etcd implementation proposal  we
created and provide feedback on that.
This helps to asses rather this approach is viable at all.

We are open to collaborate on integrating our proposed Etcd implementation
into any integration test system, to certify that all use cases works as
expected.

Balint

2018-03-30 22:21 GMT+02:00 Gwen Shapira :

> Hi,
>
> I had an offline discussion with Ismael and wanted to summarize the
> comments and questions he raised so we are all on the same page.
>
> The core issue is that this change adds a new public API. Since we already
> know that the goal for the next 1-2 years is to get rid of ZK completely.
> Do we want to go to the effort of adding (and discussing and reviewing) a
> new public API, knowing that it will be completely removed in a year? And
> since building and testing a plugin also involves effort, will anyone do it
> for something that is going to be temporary by design?
>
> Ismael, correct me if this isn't a fair representation of your concerns.
>
> Gwen
>
>
>
> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira  wrote:
>
> > Few other concerns that were raised in the previous discussion were
> around
> > the challenges both to maintainers and users in making this API pluggable
> > and how does making the interface pluggable aligns with future goals for
> > the project. At the time this was difficult to discuss because there
> wasn't
> > a concrete proposal. I want to discuss these points in the context of
> this
> > specific proposal:
> >
> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> > implementations to cover.
> > In this case: At the time, the Kafka project didn't have much
> > experience with pluggable APIs and components, so the concerns were very
> > valid. Right now Kafka has multiple pluggable components - Connectors,
> > converters, transformations, authentication protocols, authorization
> > database, coordination protocol, serializers, etc. I think that as a
> > community we gotten better at testing the interface, testing the very few
> > implementations that are included in Apache Kafka itself and allowing the
> > community to innovate and validate outside of the Kafka project. I don't
> > recall major issues either from lack of testing or from usability
> > perspective.
> >
> > 2. Problem: Users don't want to choose a consensus implementation, they
> > just don't want ZK.
> > In this case: I agree that users don't actually want to spend time
> > choosing consensus implementation and a simpler deployment model would
> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> > implementation, 99% of the users will choose to use that (a vast majority
> > uses our less-than-amazing authorization plugin), and the few that really
> > need something else for whatever reason, will be able to get what they
> > need. As Jake said, we need to face the fact that development trajectory
> of
> > ZK isn't amazing at the moment, that it is lacking features our users
> need
> > (SSL) and it will be good to allow the user community to explore
> > alternatives.
> >
> > 3. Problem: Why got to the effort of refactoring if we know we want to
> get
> > rid of ZK.
> > In this case: This change isn't huge, it doesn't rewrite large
> > portions of Kafka and it does not make the future direction any more
> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> > solution, applying it on Kafka with this KIP isn't any more challenging
> > than applying it to Kafka without this KIP. It is a step in an orthogonal
> > direction, but not opposite direction. I think that letting the perfect
> > become the enemy of the good is a repeated failure mode in this
> community.
> > Can we discuss whether this proposal is good even if there is a more
> > complex proposal that may be better? As long as they don't conflict?
> >
> > Gwen
> >
> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint 
> > wrote:
> >
> >> Thanks, for the feedback.
> >>
> >> Developing an internal consensus service inside Kafka would require a
> team
> >> dedicated to this task.
> >> We second what Flavio said in
> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f
> >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E
> >> that
> >> getting an implementation which really works and is maintainable is
> >> difficult.
> >> We think that Kafka being able to use another widely used consensus
> system
> >> beside Zookeeper its a safer and workable solution.
> >> It will be easier for users to use a consensus system with Kafka that
> they
> 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Gwen Shapira]

Hi,

I had an offline discussion with Ismael and wanted to summarize the
comments and questions he raised so we are all on the same page.

The core issue is that this change adds a new public API. Since we already
know that the goal for the next 1-2 years is to get rid of ZK completely.
Do we want to go to the effort of adding (and discussing and reviewing) a
new public API, knowing that it will be completely removed in a year? And
since building and testing a plugin also involves effort, will anyone do it
for something that is going to be temporary by design?

Ismael, correct me if this isn't a fair representation of your concerns.

Gwen



On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira  wrote:

> Few other concerns that were raised in the previous discussion were around
> the challenges both to maintainers and users in making this API pluggable
> and how does making the interface pluggable aligns with future goals for
> the project. At the time this was difficult to discuss because there wasn't
> a concrete proposal. I want to discuss these points in the context of this
> specific proposal:
>
> 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> implementations to cover.
> In this case: At the time, the Kafka project didn't have much
> experience with pluggable APIs and components, so the concerns were very
> valid. Right now Kafka has multiple pluggable components - Connectors,
> converters, transformations, authentication protocols, authorization
> database, coordination protocol, serializers, etc. I think that as a
> community we gotten better at testing the interface, testing the very few
> implementations that are included in Apache Kafka itself and allowing the
> community to innovate and validate outside of the Kafka project. I don't
> recall major issues either from lack of testing or from usability
> perspective.
>
> 2. Problem: Users don't want to choose a consensus implementation, they
> just don't want ZK.
> In this case: I agree that users don't actually want to spend time
> choosing consensus implementation and a simpler deployment model would
> serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> implementation, 99% of the users will choose to use that (a vast majority
> uses our less-than-amazing authorization plugin), and the few that really
> need something else for whatever reason, will be able to get what they
> need. As Jake said, we need to face the fact that development trajectory of
> ZK isn't amazing at the moment, that it is lacking features our users need
> (SSL) and it will be good to allow the user community to explore
> alternatives.
>
> 3. Problem: Why got to the effort of refactoring if we know we want to get
> rid of ZK.
> In this case: This change isn't huge, it doesn't rewrite large
> portions of Kafka and it does not make the future direction any more
> difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> solution, applying it on Kafka with this KIP isn't any more challenging
> than applying it to Kafka without this KIP. It is a step in an orthogonal
> direction, but not opposite direction. I think that letting the perfect
> become the enemy of the good is a repeated failure mode in this community.
> Can we discuss whether this proposal is good even if there is a more
> complex proposal that may be better? As long as they don't conflict?
>
> Gwen
>
> On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint 
> wrote:
>
>> Thanks, for the feedback.
>>
>> Developing an internal consensus service inside Kafka would require a team
>> dedicated to this task.
>> We second what Flavio said in
>> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f
>> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E
>> that
>> getting an implementation which really works and is maintainable is
>> difficult.
>> We think that Kafka being able to use another widely used consensus system
>> beside Zookeeper its a safer and workable solution.
>> It will be easier for users to use a consensus system with Kafka that they
>> are already familiar with.
>>
>>
>> The implementation found here:
>> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/
>> kafka-on-etcd/core/src/main/scala/kafka/etcd
>> is a first version of enabling Etcd in Kafka.
>> This implementation hooked in Etcd with a slight change in the existing
>> interfaces. While this implementation works its far from being complete.
>> Ideally existing interfaces should be reworked to abstract out the used
>> consensus system.
>> We opened this KIP to start a discussion and the community to have a look
>> at the initial implementation and receive feedback if this initiative is
>> viable.
>>
>> Balint
>>
>> 2018-03-29 11:23 GMT+02:00 Jakub Scholz :
>>
>> > I can understand the concerns about the plugability of Zookeeper/Etcd.
>> It
>> > would not be good for Kafka community if it splits into 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Gwen Shapira]

Few other concerns that were raised in the previous discussion were around
the challenges both to maintainers and users in making this API pluggable
and how does making the interface pluggable aligns with future goals for
the project. At the time this was difficult to discuss because there wasn't
a concrete proposal. I want to discuss these points in the context of this
specific proposal:

1. Problem: Pluggable APIs mean larger surface testing area and multiple
implementations to cover.
In this case: At the time, the Kafka project didn't have much
experience with pluggable APIs and components, so the concerns were very
valid. Right now Kafka has multiple pluggable components - Connectors,
converters, transformations, authentication protocols, authorization
database, coordination protocol, serializers, etc. I think that as a
community we gotten better at testing the interface, testing the very few
implementations that are included in Apache Kafka itself and allowing the
community to innovate and validate outside of the Kafka project. I don't
recall major issues either from lack of testing or from usability
perspective.

2. Problem: Users don't want to choose a consensus implementation, they
just don't want ZK.
In this case: I agree that users don't actually want to spend time
choosing consensus implementation and a simpler deployment model would
serve them better. IMO, if Apache Kafka ships with our well-tested ZK
implementation, 99% of the users will choose to use that (a vast majority
uses our less-than-amazing authorization plugin), and the few that really
need something else for whatever reason, will be able to get what they
need. As Jake said, we need to face the fact that development trajectory of
ZK isn't amazing at the moment, that it is lacking features our users need
(SSL) and it will be good to allow the user community to explore
alternatives.

3. Problem: Why got to the effort of refactoring if we know we want to get
rid of ZK.
In this case: This change isn't huge, it doesn't rewrite large portions
of Kafka and it does not make the future direction any more difficult. If
in 2 weeks or 2 month or 2 years we'll have a ZK-less solution, applying it
on Kafka with this KIP isn't any more challenging than applying it to Kafka
without this KIP. It is a step in an orthogonal direction, but not opposite
direction. I think that letting the perfect become the enemy of the good is
a repeated failure mode in this community. Can we discuss whether this
proposal is good even if there is a more complex proposal that may be
better? As long as they don't conflict?

Gwen

On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint 
wrote:

> Thanks, for the feedback.
>
> Developing an internal consensus service inside Kafka would require a team
> dedicated to this task.
> We second what Flavio said in
> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f7a1f1c0
> a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E
> that
> getting an implementation which really works and is maintainable is
> difficult.
> We think that Kafka being able to use another widely used consensus system
> beside Zookeeper its a safer and workable solution.
> It will be easier for users to use a consensus system with Kafka that they
> are already familiar with.
>
>
> The implementation found here:
> https://github.com/banzaicloud/apache-kafka-on-
> k8s/tree/kafka-on-etcd/core/src/main/scala/kafka/etcd
> is a first version of enabling Etcd in Kafka.
> This implementation hooked in Etcd with a slight change in the existing
> interfaces. While this implementation works its far from being complete.
> Ideally existing interfaces should be reworked to abstract out the used
> consensus system.
> We opened this KIP to start a discussion and the community to have a look
> at the initial implementation and receive feedback if this initiative is
> viable.
>
> Balint
>
> 2018-03-29 11:23 GMT+02:00 Jakub Scholz :
>
> > I can understand the concerns about the plugability of Zookeeper/Etcd. It
> > would not be good for Kafka community if it splits into several groups
> > using different implementations.
> >
> > On the other hand, Zookeeper development seems to be a bit stalled. So
> > maybe there should be at least a discussion whether it makes sense to
> > replace Zookeeper with something like Etcd or not.
> >
> > JAkub
> >
> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> > >
> > > Here is the link to the KIP:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> > >
> > > Looking forward to the discussion.
> > >
> > > Thanks,
> > > Balint
> > >
> >
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Thanks, for the feedback.

Developing an internal consensus service inside Kafka would require a team
dedicated to this task.
We second what Flavio said in
https://lists.apache.org/thread.html/24ae56e073104c4531cf64f7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E
that
getting an implementation which really works and is maintainable is
difficult.
We think that Kafka being able to use another widely used consensus system
beside Zookeeper its a safer and workable solution.
It will be easier for users to use a consensus system with Kafka that they
are already familiar with.


The implementation found here:
https://github.com/banzaicloud/apache-kafka-on-k8s/tree/kafka-on-etcd/core/src/main/scala/kafka/etcd
is a first version of enabling Etcd in Kafka.
This implementation hooked in Etcd with a slight change in the existing
interfaces. While this implementation works its far from being complete.
Ideally existing interfaces should be reworked to abstract out the used
consensus system.
We opened this KIP to start a discussion and the community to have a look
at the initial implementation and receive feedback if this initiative is
viable.

Balint

2018-03-29 11:23 GMT+02:00 Jakub Scholz :

> I can understand the concerns about the plugability of Zookeeper/Etcd. It
> would not be good for Kafka community if it splits into several groups
> using different implementations.
>
> On the other hand, Zookeeper development seems to be a bit stalled. So
> maybe there should be at least a discussion whether it makes sense to
> replace Zookeeper with something like Etcd or not.
>
> JAkub
>
> On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint 
> wrote:
>
> > Hi all,
> >
> > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> >
> > Here is the link to the KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >
> > Looking forward to the discussion.
> >
> > Thanks,
> > Balint
> >
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Jakub Scholz]

I can understand the concerns about the plugability of Zookeeper/Etcd. It
would not be good for Kafka community if it splits into several groups
using different implementations.

On the other hand, Zookeeper development seems to be a bit stalled. So
maybe there should be at least a discussion whether it makes sense to
replace Zookeeper with something like Etcd or not.

JAkub

On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint 
wrote:

> Hi all,
>
> I have created KIP-273: Kafka to support using ETCD beside Zookeeper
>
> Here is the link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
>
> Looking forward to the discussion.
>
> Thanks,
> Balint
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Tom Lee]

Not sure if this is entirely relevant, but figure folks might be
interested: it's early days for us, but we're currently running Kafka+ZK in
AWS on Kubernetes StatefulSets with the magic annotation to publish "not
ready" addresses (can't recall what it is off the top of my head). It seems
stable so far and we can deploy & redeploy without issue. The jury is still
out on whether running Kafka+ZK on Kubernetes is *advisable*, but I can
talk about what we had to do to get to where we are.

The "worst" thing we had to do was patch the ZK client libs on the Kafka
side to replace the default HostProvider implementation with a special
"kubernetes-friendly" HostProvider to force DNS lookups: default behavior
of StaticHostProvider is to resolve once & cache IPs at the time the client
is created. We also had to configure our JVM DNS TTLs to essentially
nothing. We were bitten by KAFKA-2729 in one of our testing environments
for reasons I still don't fully understand, so we're chomping at the bit
for 1.1.0. :)

Otherwise ... so far so good -- but again, very early days.

Also maybe interesting is that the ZK 3.5 client libs allow pluggable
HostProviders, which will give you folks (Kafka devs) more control over the
ZK DNS lookups without needing to do crazy patching stuff like we did. All
the necessary mechanics to do this are in ZK 3.4.x, but not exposed at the
API level. I've contemplated sending a patch upstream to see if they'd
consider allowing a system property to control the HostProvider
implementation in 3.4 as sort of a poor-man's backport -- then we wouldn't
need to patch the client libs to make things play nice with Kubernetes. No
idea what the temperature would be on that, though.

Anyway: don't consider this an endorsement of the idea and I'm not familiar
enough with etcd to comment on whether ZK is remarkably more difficult to
manage as the KIP seems to imply, just wanted to comment on our experience
so far.

Cheers,
Tom


On Wed, Mar 28, 2018 at 2:00 PM, Ismael Juma  wrote:

> Here's a link to the previous discussion:
>
> https://lists.apache.org/thread.html/2bc187040051008452b40b313db06b
> 476c248ef7a5ed7529afe7b118@1448997154@%3Cdev.kafka.apache.org%3E
>
> Ismael
>
> On Wed, Mar 28, 2018 at 10:40 AM, Ismael Juma  wrote:
>
> > Hi Gwen,
> >
> > I don't think the reasons why a pluggable metastore is not desirable have
> > changed. My suggestion is that the KIP should try to address the concerns
> > raised previously as part of the proposal.
> >
> > Ismael
> >
> > On Wed, Mar 28, 2018 at 10:24 AM, Gwen Shapira 
> wrote:
> >
> >> While I'm not in favor of the proposal, I want to point out that the
> >> ecosystem changed quite a bit since KIP-30 was first proposed.
> Kubernetes
> >> deployments are far more common now and are growing in popularity, and
> the
> >> problem in deployment, discovery and management that ZK poses is
> therefore
> >> more relevant now than it was at the time. There are reasons for the
> >> community to change its collective mind even if the objections are still
> >> valid.
> >>
> >> Since the KIP doesn't include the etcd implementation, the proposal
> looks
> >> like very simple refactoring. Of course, the big change is a new public
> >> API. But it's difficult to judge from the KIP if the API is a good one
> >> because it is built to 100% match the one implementation we have. I'm
> >> curious if the plan includes contributing the Etcd module to Apache
> Kafka?
> >>
> >>
> >> On Wed, Mar 28, 2018 at 9:54 AM, Ismael Juma  wrote:
> >>
> >> > Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
> >> > brokers to have plug-able consensus and meta data storage sub systems"
> >> and
> >> > the community was not in favour. Have you considered the points
> >> discussed
> >> > then?
> >> >
> >> > Ismael
> >> >
> >> > On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint <
> molnarcsi...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> >> > >
> >> > > Here is the link to the KIP:
> >> > >
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >> > >
> >> > > Looking forward to the discussion.
> >> > >
> >> > > Thanks,
> >> > > Balint
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> *Gwen Shapira*
> >> Product Manager | Confluent
> >> 650.450.2760 | @gwenshap
> >> Follow us: Twitter  | blog
> >> 
> >>
> >
> >
>



-- 
*Tom Lee */ http://tomlee.co / @tglee 

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Ismael Juma]

Here's a link to the previous discussion:

https://lists.apache.org/thread.html/2bc187040051008452b40b313db06b476c248ef7a5ed7529afe7b118@1448997154@%3Cdev.kafka.apache.org%3E

Ismael

On Wed, Mar 28, 2018 at 10:40 AM, Ismael Juma  wrote:

> Hi Gwen,
>
> I don't think the reasons why a pluggable metastore is not desirable have
> changed. My suggestion is that the KIP should try to address the concerns
> raised previously as part of the proposal.
>
> Ismael
>
> On Wed, Mar 28, 2018 at 10:24 AM, Gwen Shapira  wrote:
>
>> While I'm not in favor of the proposal, I want to point out that the
>> ecosystem changed quite a bit since KIP-30 was first proposed. Kubernetes
>> deployments are far more common now and are growing in popularity, and the
>> problem in deployment, discovery and management that ZK poses is therefore
>> more relevant now than it was at the time. There are reasons for the
>> community to change its collective mind even if the objections are still
>> valid.
>>
>> Since the KIP doesn't include the etcd implementation, the proposal looks
>> like very simple refactoring. Of course, the big change is a new public
>> API. But it's difficult to judge from the KIP if the API is a good one
>> because it is built to 100% match the one implementation we have. I'm
>> curious if the plan includes contributing the Etcd module to Apache Kafka?
>>
>>
>> On Wed, Mar 28, 2018 at 9:54 AM, Ismael Juma  wrote:
>>
>> > Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
>> > brokers to have plug-able consensus and meta data storage sub systems"
>> and
>> > the community was not in favour. Have you considered the points
>> discussed
>> > then?
>> >
>> > Ismael
>> >
>> > On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint 
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
>> > >
>> > > Here is the link to the KIP:
>> > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
>> > >
>> > > Looking forward to the discussion.
>> > >
>> > > Thanks,
>> > > Balint
>> > >
>> >
>>
>>
>>
>> --
>> *Gwen Shapira*
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter  | blog
>> 
>>
>
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Ismael Juma]

Hi Gwen,

I don't think the reasons why a pluggable metastore is not desirable have
changed. My suggestion is that the KIP should try to address the concerns
raised previously as part of the proposal.

Ismael

On Wed, Mar 28, 2018 at 10:24 AM, Gwen Shapira  wrote:

> While I'm not in favor of the proposal, I want to point out that the
> ecosystem changed quite a bit since KIP-30 was first proposed. Kubernetes
> deployments are far more common now and are growing in popularity, and the
> problem in deployment, discovery and management that ZK poses is therefore
> more relevant now than it was at the time. There are reasons for the
> community to change its collective mind even if the objections are still
> valid.
>
> Since the KIP doesn't include the etcd implementation, the proposal looks
> like very simple refactoring. Of course, the big change is a new public
> API. But it's difficult to judge from the KIP if the API is a good one
> because it is built to 100% match the one implementation we have. I'm
> curious if the plan includes contributing the Etcd module to Apache Kafka?
>
>
> On Wed, Mar 28, 2018 at 9:54 AM, Ismael Juma  wrote:
>
> > Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
> > brokers to have plug-able consensus and meta data storage sub systems"
> and
> > the community was not in favour. Have you considered the points discussed
> > then?
> >
> > Ismael
> >
> > On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> > >
> > > Here is the link to the KIP:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> > >
> > > Looking forward to the discussion.
> > >
> > > Thanks,
> > > Balint
> > >
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Gwen Shapira]

While I'm not in favor of the proposal, I want to point out that the
ecosystem changed quite a bit since KIP-30 was first proposed. Kubernetes
deployments are far more common now and are growing in popularity, and the
problem in deployment, discovery and management that ZK poses is therefore
more relevant now than it was at the time. There are reasons for the
community to change its collective mind even if the objections are still
valid.

Since the KIP doesn't include the etcd implementation, the proposal looks
like very simple refactoring. Of course, the big change is a new public
API. But it's difficult to judge from the KIP if the API is a good one
because it is built to 100% match the one implementation we have. I'm
curious if the plan includes contributing the Etcd module to Apache Kafka?


On Wed, Mar 28, 2018 at 9:54 AM, Ismael Juma  wrote:

> Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
> brokers to have plug-able consensus and meta data storage sub systems" and
> the community was not in favour. Have you considered the points discussed
> then?
>
> Ismael
>
> On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint 
> wrote:
>
> > Hi all,
> >
> > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> >
> > Here is the link to the KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >
> > Looking forward to the discussion.
> >
> > Thanks,
> > Balint
> >
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Ismael Juma]

Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
brokers to have plug-able consensus and meta data storage sub systems" and
the community was not in favour. Have you considered the points discussed
then?

Ismael

On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint 
wrote:

> Hi all,
>
> I have created KIP-273: Kafka to support using ETCD beside Zookeeper
>
> Here is the link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
>
> Looking forward to the discussion.
>
> Thanks,
> Balint
>

[DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi all,

I have created KIP-273: Kafka to support using ETCD beside Zookeeper

Here is the link to the KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper

Looking forward to the discussion.

Thanks,
Balint