Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi Jakub,

Yes I tried it and as I saw in various forums it works well for Kafka as
well, but because it mimics the Zookeeper API it creates quite many
metadata related value.
Kafka uses only a subset from the Zookeeper API, so those values can be
omitted by implementing Kafka Zookeeper calls in a more Etcd way.
Also in the future I hope we can replace the Zookeeper specific interface
inside Kafka to a more generic one.

Regards
Balint

Jakub Scholz <ja...@scholz.cz> ezt írta (időpont: 2018. máj. 17., Cs,
22:19):

> Hi Balint,
>
> Out of curiosity - have you ever tried this project
> https://github.com/coreos/zetcd which tries to provide Zookeeper API for
> Etcd?
>
> Thanks & Regards
> Jakub
>
> On Wed, Mar 28, 2018 at 6:18 PM Molnár Bálint <molnarcsi...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> >
> > Here is the link to the KIP:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >
> > Looking forward to the discussion.
> >
> > Thanks,
> > Balint
> >
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi Colin,

Do you have a rough estimate on that?

Thanks,
Balint

Colin McCabe <cmcc...@apache.org> ezt írta (időpont: 2018. máj. 9., Sze,
19:53):

> Hi Molnar,
>
> The points Ismael brought up earlier (and that were brought up on KIP-30)
> are still relevant here.  As Ismael said, the goal is to get rid of
> external dependencies here.   We're going to post more about this soon
> (sorry for the delay)
>
> thanks,
> Colin
>
>
> On Wed, May 9, 2018, at 07:29, Molnár Bálint wrote:
> > Hi,
> > I just rebased the Etcd implementation proposal on trunk. Pinging to see
> if
> > anyone has feedback on my questions from my previous email.
> >
> > Molnár Bálint <molnarcsi...@gmail.com> ezt írta (időpont: 2018. ápr. 4.,
> > Sze, 10:08):
> >
> > > Hi,
> > > Thanks again for the feedback.
> > >
> > > Is there already ongoing work for having an own consensus
> implementation
> > > within Kafka?
> > > If that work haven't started yet, we think there is value in having an
> > > interim solution, that allows the use of another consensus system
> besides
> > > Zookeeper.
> > >
> > > We ask the community to take a look at the Etcd implementation proposal
> > > we created and provide feedback on that.
> > > This helps to asses rather this approach is viable at all.
> > >
> > > We are open to collaborate on integrating our proposed Etcd
> implementation
> > > into any integration test system, to certify that all use cases works
> as
> > > expected.
> > >
> > > Balint
> > >
> > > 2018-03-30 22:21 GMT+02:00 Gwen Shapira <g...@confluent.io>:
> > >
> > >> Hi,
> > >>
> > >> I had an offline discussion with Ismael and wanted to summarize the
> > >> comments and questions he raised so we are all on the same page.
> > >>
> > >> The core issue is that this change adds a new public API. Since we
> already
> > >> know that the goal for the next 1-2 years is to get rid of ZK
> completely.
> > >> Do we want to go to the effort of adding (and discussing and
> reviewing) a
> > >> new public API, knowing that it will be completely removed in a year?
> And
> > >> since building and testing a plugin also involves effort, will anyone
> do
> > >> it
> > >> for something that is going to be temporary by design?
> > >>
> > >> Ismael, correct me if this isn't a fair representation of your
> concerns.
> > >>
> > >> Gwen
> > >>
> > >>
> > >>
> > >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io>
> wrote:
> > >>
> > >> > Few other concerns that were raised in the previous discussion were
> > >> around
> > >> > the challenges both to maintainers and users in making this API
> > >> pluggable
> > >> > and how does making the interface pluggable aligns with future
> goals for
> > >> > the project. At the time this was difficult to discuss because there
> > >> wasn't
> > >> > a concrete proposal. I want to discuss these points in the context
> of
> > >> this
> > >> > specific proposal:
> > >> >
> > >> > 1. Problem: Pluggable APIs mean larger surface testing area and
> multiple
> > >> > implementations to cover.
> > >> > In this case: At the time, the Kafka project didn't have much
> > >> > experience with pluggable APIs and components, so the concerns were
> very
> > >> > valid. Right now Kafka has multiple pluggable components -
> Connectors,
> > >> > converters, transformations, authentication protocols, authorization
> > >> > database, coordination protocol, serializers, etc. I think that as a
> > >> > community we gotten better at testing the interface, testing the
> very
> > >> few
> > >> > implementations that are included in Apache Kafka itself and
> allowing
> > >> the
> > >> > community to innovate and validate outside of the Kafka project. I
> don't
> > >> > recall major issues either from lack of testing or from usability
> > >> > perspective.
> > >> >
> > >> > 2. Problem: Users don't want to choose a consensus implementation,
> they
> > >> > just don't want ZK.
> > >> > In this case: I agree that u

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi,
I just rebased the Etcd implementation proposal on trunk. Pinging to see if
anyone has feedback on my questions from my previous email.

Molnár Bálint <molnarcsi...@gmail.com> ezt írta (időpont: 2018. ápr. 4.,
Sze, 10:08):

> Hi,
> Thanks again for the feedback.
>
> Is there already ongoing work for having an own consensus implementation
> within Kafka?
> If that work haven't started yet, we think there is value in having an
> interim solution, that allows the use of another consensus system besides
> Zookeeper.
>
> We ask the community to take a look at the Etcd implementation proposal
> we created and provide feedback on that.
> This helps to asses rather this approach is viable at all.
>
> We are open to collaborate on integrating our proposed Etcd implementation
> into any integration test system, to certify that all use cases works as
> expected.
>
> Balint
>
> 2018-03-30 22:21 GMT+02:00 Gwen Shapira <g...@confluent.io>:
>
>> Hi,
>>
>> I had an offline discussion with Ismael and wanted to summarize the
>> comments and questions he raised so we are all on the same page.
>>
>> The core issue is that this change adds a new public API. Since we already
>> know that the goal for the next 1-2 years is to get rid of ZK completely.
>> Do we want to go to the effort of adding (and discussing and reviewing) a
>> new public API, knowing that it will be completely removed in a year? And
>> since building and testing a plugin also involves effort, will anyone do
>> it
>> for something that is going to be temporary by design?
>>
>> Ismael, correct me if this isn't a fair representation of your concerns.
>>
>> Gwen
>>
>>
>>
>> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io> wrote:
>>
>> > Few other concerns that were raised in the previous discussion were
>> around
>> > the challenges both to maintainers and users in making this API
>> pluggable
>> > and how does making the interface pluggable aligns with future goals for
>> > the project. At the time this was difficult to discuss because there
>> wasn't
>> > a concrete proposal. I want to discuss these points in the context of
>> this
>> > specific proposal:
>> >
>> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
>> > implementations to cover.
>> > In this case: At the time, the Kafka project didn't have much
>> > experience with pluggable APIs and components, so the concerns were very
>> > valid. Right now Kafka has multiple pluggable components - Connectors,
>> > converters, transformations, authentication protocols, authorization
>> > database, coordination protocol, serializers, etc. I think that as a
>> > community we gotten better at testing the interface, testing the very
>> few
>> > implementations that are included in Apache Kafka itself and allowing
>> the
>> > community to innovate and validate outside of the Kafka project. I don't
>> > recall major issues either from lack of testing or from usability
>> > perspective.
>> >
>> > 2. Problem: Users don't want to choose a consensus implementation, they
>> > just don't want ZK.
>> > In this case: I agree that users don't actually want to spend time
>> > choosing consensus implementation and a simpler deployment model would
>> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
>> > implementation, 99% of the users will choose to use that (a vast
>> majority
>> > uses our less-than-amazing authorization plugin), and the few that
>> really
>> > need something else for whatever reason, will be able to get what they
>> > need. As Jake said, we need to face the fact that development
>> trajectory of
>> > ZK isn't amazing at the moment, that it is lacking features our users
>> need
>> > (SSL) and it will be good to allow the user community to explore
>> > alternatives.
>> >
>> > 3. Problem: Why got to the effort of refactoring if we know we want to
>> get
>> > rid of ZK.
>> > In this case: This change isn't huge, it doesn't rewrite large
>> > portions of Kafka and it does not make the future direction any more
>> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
>> > solution, applying it on Kafka with this KIP isn't any more challenging
>> > than applying it to Kafka without this KIP. It is a step in an
>> orthogonal
>> > direction, but not oppo

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi,
Thanks again for the feedback.

Is there already ongoing work for having an own consensus implementation
within Kafka?
If that work haven't started yet, we think there is value in having an
interim solution, that allows the use of another consensus system besides
Zookeeper.

We ask the community to take a look at the Etcd implementation proposal  we
created and provide feedback on that.
This helps to asses rather this approach is viable at all.

We are open to collaborate on integrating our proposed Etcd implementation
into any integration test system, to certify that all use cases works as
expected.

Balint

2018-03-30 22:21 GMT+02:00 Gwen Shapira <g...@confluent.io>:

> Hi,
>
> I had an offline discussion with Ismael and wanted to summarize the
> comments and questions he raised so we are all on the same page.
>
> The core issue is that this change adds a new public API. Since we already
> know that the goal for the next 1-2 years is to get rid of ZK completely.
> Do we want to go to the effort of adding (and discussing and reviewing) a
> new public API, knowing that it will be completely removed in a year? And
> since building and testing a plugin also involves effort, will anyone do it
> for something that is going to be temporary by design?
>
> Ismael, correct me if this isn't a fair representation of your concerns.
>
> Gwen
>
>
>
> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io> wrote:
>
> > Few other concerns that were raised in the previous discussion were
> around
> > the challenges both to maintainers and users in making this API pluggable
> > and how does making the interface pluggable aligns with future goals for
> > the project. At the time this was difficult to discuss because there
> wasn't
> > a concrete proposal. I want to discuss these points in the context of
> this
> > specific proposal:
> >
> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> > implementations to cover.
> > In this case: At the time, the Kafka project didn't have much
> > experience with pluggable APIs and components, so the concerns were very
> > valid. Right now Kafka has multiple pluggable components - Connectors,
> > converters, transformations, authentication protocols, authorization
> > database, coordination protocol, serializers, etc. I think that as a
> > community we gotten better at testing the interface, testing the very few
> > implementations that are included in Apache Kafka itself and allowing the
> > community to innovate and validate outside of the Kafka project. I don't
> > recall major issues either from lack of testing or from usability
> > perspective.
> >
> > 2. Problem: Users don't want to choose a consensus implementation, they
> > just don't want ZK.
> > In this case: I agree that users don't actually want to spend time
> > choosing consensus implementation and a simpler deployment model would
> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> > implementation, 99% of the users will choose to use that (a vast majority
> > uses our less-than-amazing authorization plugin), and the few that really
> > need something else for whatever reason, will be able to get what they
> > need. As Jake said, we need to face the fact that development trajectory
> of
> > ZK isn't amazing at the moment, that it is lacking features our users
> need
> > (SSL) and it will be good to allow the user community to explore
> > alternatives.
> >
> > 3. Problem: Why got to the effort of refactoring if we know we want to
> get
> > rid of ZK.
> > In this case: This change isn't huge, it doesn't rewrite large
> > portions of Kafka and it does not make the future direction any more
> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> > solution, applying it on Kafka with this KIP isn't any more challenging
> > than applying it to Kafka without this KIP. It is a step in an orthogonal
> > direction, but not opposite direction. I think that letting the perfect
> > become the enemy of the good is a repeated failure mode in this
> community.
> > Can we discuss whether this proposal is good even if there is a more
> > complex proposal that may be better? As long as they don't conflict?
> >
> > Gwen
> >
> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint <molnarcsi...@gmail.com>
> > wrote:
> >
> >> Thanks, for the feedback.
> >>
> >> Developing an internal consensus service inside Kafka would require a
> team
> >> dedicated to this task.
> >> We second what Flavio said in
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Thanks, for the feedback.

Developing an internal consensus service inside Kafka would require a team
dedicated to this task.
We second what Flavio said in
https://lists.apache.org/thread.html/24ae56e073104c4531cf64f7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E
that
getting an implementation which really works and is maintainable is
difficult.
We think that Kafka being able to use another widely used consensus system
beside Zookeeper its a safer and workable solution.
It will be easier for users to use a consensus system with Kafka that they
are already familiar with.


The implementation found here:
https://github.com/banzaicloud/apache-kafka-on-k8s/tree/kafka-on-etcd/core/src/main/scala/kafka/etcd
is a first version of enabling Etcd in Kafka.
This implementation hooked in Etcd with a slight change in the existing
interfaces. While this implementation works its far from being complete.
Ideally existing interfaces should be reworked to abstract out the used
consensus system.
We opened this KIP to start a discussion and the community to have a look
at the initial implementation and receive feedback if this initiative is
viable.

Balint

2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>:

> I can understand the concerns about the plugability of Zookeeper/Etcd. It
> would not be good for Kafka community if it splits into several groups
> using different implementations.
>
> On the other hand, Zookeeper development seems to be a bit stalled. So
> maybe there should be at least a discussion whether it makes sense to
> replace Zookeeper with something like Etcd or not.
>
> JAkub
>
> On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint <molnarcsi...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> >
> > Here is the link to the KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >
> > Looking forward to the discussion.
> >
> > Thanks,
> > Balint
> >
>

[DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Molnár Bálint]

Hi all,

I have created KIP-273: Kafka to support using ETCD beside Zookeeper

Here is the link to the KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-273+-+Kafka+to+support+using+ETCD+beside+Zookeeper

Looking forward to the discussion.

Thanks,
Balint

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

[Molnár Bálint]

+1 (non-binding)

2017-08-31 10:33 GMT+02:00 Manikumar :

> +1 (non-binding)
>
> On Thu, Aug 31, 2017 at 1:53 PM, Ismael Juma  wrote:
>
> > Thanks for the KIP, +1 (binding).
> >
> > Ismael
> >
> > On 31 Aug 2017 8:38 am, "Attila Kreiner"  wrote:
> >
> > Hi All,
> >
> > Thx for the comments, I pretty much see a consensus here. So I'd like to
> > start the vote for:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 191%3A+KafkaConsumer.
> > subscribe%28%29+overload+that+takes+just+Pattern
> >
> > Cheers,
> > Attila
> >
>

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

[Molnár Bálint]

Congrats, Ismael:)

2017-07-06 2:57 GMT+02:00 Matthias J. Sax :

> Congrats!
>
> On 7/5/17 3:42 PM, Roger Hoover wrote:
> > Well deserved, indeed!  Congrats, Ismael.
> >
> > On Wed, Jul 5, 2017 at 3:24 PM, Damian Guy  wrote:
> >
> >> Congratulations Ismael! Very well deserved.
> >> Cheers,
> >> Damian
> >> On Wed, 5 Jul 2017 at 22:54, Dong Lin  wrote:
> >>
> >>> Congratulations Ismael!
> >>>
> >>> On Wed, Jul 5, 2017 at 1:55 PM, Jun Rao  wrote:
> >>>
>  Hi, Everyone,
> 
>  Ismael Juma has been active in the Kafka community since he became
>  a Kafka committer about a year ago. I am glad to announce that Ismael
> >> is
>  now a member of Kafka PMC.
> 
>  Congratulations, Ismael!
> 
>  Jun
> 
> >>>
> >>
> >
>
>

Re: [ANNOUNCE] New committer: Damian Guy

[Molnár Bálint]

Congrats, Damien!

2017-06-12 8:44 GMT+02:00 Rajini Sivaram :

> Congratulations, Damian!
>
> On Sat, Jun 10, 2017 at 12:15 PM, Mickael Maison  >
> wrote:
>
> > Congrats Damian!
> >
> > On Sat, Jun 10, 2017 at 8:46 AM, Damian Guy 
> wrote:
> > > Thanks everyone. Looking forward to making many more contributions
> > > On Sat, 10 Jun 2017 at 02:46, Joe Stein  wrote:
> > >
> > >> Congrats!
> > >>
> > >>
> > >> ~ Joe Stein
> > >>
> > >> On Fri, Jun 9, 2017 at 6:49 PM, Neha Narkhede 
> > wrote:
> > >>
> > >> > Well deserved. Congratulations Damian!
> > >> >
> > >> > On Fri, Jun 9, 2017 at 1:34 PM Guozhang Wang 
> > wrote:
> > >> >
> > >> > > Hello all,
> > >> > >
> > >> > >
> > >> > > The PMC of Apache Kafka is pleased to announce that we have
> invited
> > >> > Damian
> > >> > > Guy as a committer to the project.
> > >> > >
> > >> > > Damian has made tremendous contributions to Kafka. He has not only
> > >> > > contributed a lot into the Streams api, but have also been
> involved
> > in
> > >> > many
> > >> > > other areas like the producer and consumer clients, broker-side
> > >> > > coordinators (group coordinator and the ongoing transaction
> > >> coordinator).
> > >> > > He has contributed more than 100 patches so far, and have been
> > driving
> > >> > on 6
> > >> > > KIP contributions.
> > >> > >
> > >> > > More importantly, Damian has been a very prolific reviewer on open
> > PRs
> > >> > and
> > >> > > has been actively participating on community activities such as
> > email
> > >> > lists
> > >> > > and slack overflow questions. Through his code contributions and
> > >> reviews,
> > >> > > Damian has demonstrated good judgement on system design and code
> > >> > qualities,
> > >> > > especially on thorough unit test coverages. We believe he will
> make
> > a
> > >> > great
> > >> > > addition to the committers of the community.
> > >> > >
> > >> > >
> > >> > > Thank you for your contributions, Damian!
> > >> > >
> > >> > >
> > >> > > -- Guozhang, on behalf of the Apache Kafka PMC
> > >> > >
> > >> > --
> > >> > Thanks,
> > >> > Neha
> > >> >
> > >>
> >
>

Re: [ANNOUNCE] New committer: Rajini Sivaram

[Molnár Bálint]

Congrats, Rajini:)

2017-04-25 13:31 GMT+02:00 Eno Thereska :

> Congrats!
>
> Eno
> > On Apr 25, 2017, at 12:17 PM, Rajini Sivaram 
> wrote:
> >
> > Thanks everyone!
> >
> > It has been a pleasure working with all of you in the Kafka community.
> Many
> > thanks to the PMC for this exciting opportunity.
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Apr 25, 2017 at 10:51 AM, Damian Guy 
> wrote:
> >
> >> Congrats
> >> On Tue, 25 Apr 2017 at 09:57, Mickael Maison 
> >> wrote:
> >>
> >>> Congratulation Rajini !
> >>> Great news
> >>>
> >>> On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar 
> >> wrote:
>  Congratulations Rajini !!!
>  Well deserved
>  --
>  Edoardo Comar
>  IBM MessageHub
>  eco...@uk.ibm.com
>  IBM UK Ltd, Hursley Park, SO21 2JN
> 
>  IBM United Kingdom Limited Registered in England and Wales with number
>  741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants.
> >>> PO6
>  3AU
> 
> 
> 
>  From:   Gwen Shapira 
>  To: dev@kafka.apache.org, Users ,
>  priv...@kafka.apache.org
>  Date:   24/04/2017 22:07
>  Subject:[ANNOUNCE] New committer: Rajini Sivaram
> 
> 
> 
>  The PMC for Apache Kafka has invited Rajini Sivaram as a committer and
> >> we
>  are pleased to announce that she has accepted!
> 
>  Rajini contributed 83 patches, 8 KIPs (all security and quota
>  improvements) and a significant number of reviews. She is also on the
>  conference committee for Kafka Summit, where she helped select content
>  for our community event. Through her contributions she's shown good
>  judgement, good coding skills, willingness to work with the community
> >> on
>  finding the best
>  solutions and very consistent follow through on her work.
> 
>  Thank you for your contributions, Rajini! Looking forward to many more
> >> :)
> 
>  Gwen, for the Apache Kafka PMC
> 
> 
> 
>  Unless stated otherwise above:
>  IBM United Kingdom Limited - Registered in England and Wales with
> >> number
>  741598.
>  Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> >>> 3AU
> >>>
> >>
>
>

Re: Wiki edit permission

[Molnár Bálint]

Thanks, Guozhang:)

2017-03-07 16:03 GMT+01:00 Guozhang Wang <wangg...@gmail.com>:

> Done.
>
> Cheers,
> Guozhang
>
> On Tue, Mar 7, 2017 at 2:59 AM, Molnár Bálint <molnarcsi...@gmail.com>
> wrote:
>
> > Just a kindly remainder.
> >
> > Thanks,
> > Balint
> >
> > 2017-03-03 8:44 GMT+01:00 Molnár Bálint <molnarcsi...@gmail.com>:
> >
> > > Hi,
> > >
> > > I would like to create a KIP, please add create permission to the wiki
> > > page.
> > > My username is baluchicken.
> > >
> > > Thanks
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: Wiki edit permission

[Molnár Bálint]

Just a kindly remainder.

Thanks,
Balint

2017-03-03 8:44 GMT+01:00 Molnár Bálint <molnarcsi...@gmail.com>:

> Hi,
>
> I would like to create a KIP, please add create permission to the wiki
> page.
> My username is baluchicken.
>
> Thanks
>

Re: Allow to replace Zookeeper with different Coordination Service (etcd/consul)

[Molnár Bálint]

Thanks, for pointing me to the right direction:). I will take a look on
this KIP.

2017-03-06 16:48 GMT+01:00 Ofir Manor <ofir.ma...@equalum.io>:

> I suggest you check KIP-30 and the extensive discussion about it in the
> mailing list  from around December 2015 called "[DISCUSS] KIP-30 Allow for
> brokers to have plug-able consensus and meta data storage sub systems"
> If I remember correctly, it ran into some objections, as the existing
> commiters thought at the time that ZK was significantly more reliable, so
> it was not worth the effort to add inferior options.
> I personally think nowadays, when a lot of other critical cluster infra
> relies on these coordination services anyway, this KIP makes a lot of
> sense. The current dependency of ZK creates a large, unneeded operational
> overhead for those who have already deployed and relies on etcd/consul for
> the rest of their stack (including other stateful services).
> Just my two cents,
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Mon, Mar 6, 2017 at 12:15 PM, Alexander Binzberger <
> alexander.binzber...@wingcon.com> wrote:
>
> > I would also be interested in this. (etcd)
> >
> >
> > Am 02.03.2017 um 12:24 schrieb Molnár Bálint:
> >
> >> Hi,
> >>
> >> I was wonderring to refactor Kafka core code to be able to use different
> >> Coordination Service than Zookeeper. I know I will need to create a KIP
> >> for
> >> that.
> >> I think the first part of this task to refactor the classes which are
> >> using
> >> the ZkUtil methods to use a zookeeper independent trait instead.
> >> After that I think it will be possible to replace Zookeeper with
> >> etcd/Consul or even with a Raft implementation.
> >> Even without additional implementation it would help to test the code
> >> without starting an embedded zookeeper.
> >> I have already started to implement a POC and it seems doable, even if
> >> it's
> >> not a small patch.
> >>
> >> Balint
> >>
> >>
> > --
> > Alexander Binzberger
> > System Designer - WINGcon AG
> > Tel. +49 7543 966-119
> >
> > Sitz der Gesellschaft: Langenargen
> > Registergericht: ULM, HRB 734260
> > USt-Id.: DE232931635, WEEE-Id.: DE74015979
> > Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul (Stellvertreter),
> > Tobias Treß
> > Aufsichtsrat: Jürgen Maucher (Vorsitz), Andreas Paul (Stellvertreter),
> > Martin Sauter
> >
> >
>

Wiki edit permission

[Molnár Bálint]

Hi,

I would like to create a KIP, please add create permission to the wiki
page.
My username is baluchicken.

Thanks

Allow to replace Zookeeper with different Coordination Service (etcd/consul)

[Molnár Bálint]

Hi,

I was wonderring to refactor Kafka core code to be able to use different
Coordination Service than Zookeeper. I know I will need to create a KIP for
that.
I think the first part of this task to refactor the classes which are using
the ZkUtil methods to use a zookeeper independent trait instead.
After that I think it will be possible to replace Zookeeper with
etcd/Consul or even with a Raft implementation.
Even without additional implementation it would help to test the code
without starting an embedded zookeeper.
I have already started to implement a POC and it seems doable, even if it's
not a small patch.

Balint

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Molnár Bálint]

Thanks Ismael, +1 for 0.11.0.0

2017-02-28 15:20 GMT+01:00 Ofir Manor :

> Thanks Ismael, makes perfect sense :)
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Tue, Feb 28, 2017 at 1:02 PM, Ismael Juma  wrote:
>
> > Hi Ofir,
> >
> > Thanks for the feedback. This will be the first release with
> exactly-once.
> > I think we should give ourselves at least one
> stabilisation/polish/feedback
> > cycle before we go for 1.0. :)
> >
> > Ismael
> >
> > On Tue, Feb 28, 2017 at 10:55 AM, Ofir Manor 
> > wrote:
> >
> > > This is definitely a major release, looks already quite exciting...
> > > I don't want to start a bike-shading argument, but a few people have
> told
> > > me in the past year that once exactly-once delivery lands, Kafka would
> > > likely bump to 1.0. I do like it, but don't feel strongly either way.
> > >
> > > Ofir Manor
> > >
> > > Co-Founder & CTO | Equalum
> > >
> > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
> > >
> > > On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma 
> wrote:
> > >
> > > > Hi Vahid,
> > > >
> > > > Sure, I've added KIP-54. I thought I had done it, sorry for the
> > > oversight.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com> wrote:
> > > >
> > > > > +1 on 0.11.0.0.
> > > > >
> > > > > Can we also include KIP-54 to the list?
> > > > > The PR for this KIP is ready for review.
> > > > >
> > > > > Thanks.
> > > > > --Vahid
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From:   Ismael Juma 
> > > > > To: dev@kafka.apache.org
> > > > > Date:   02/27/2017 07:47 PM
> > > > > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > > > > Sent by:isma...@gmail.com
> > > > >
> > > > >
> > > > >
> > > > > Hi all,
> > > > >
> > > > > With 0.10.2.0 out of the way, I would like to volunteer to be the
> > > release
> > > > > manager for our next time-based release. See
> > > https://cwiki.apache.org/c
> > > > > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed
> > previous
> > > > > communication on time-based releases or need a reminder.
> > > > >
> > > > > I put together a draft release plan with June 2017 as the release
> > month
> > > > > (as
> > > > > previously agreed) and a list of KIPs that have already been voted:
> > > > >
> > > > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=68716876
> > > > >  > > > action?pageId=68716876
> > > > > >*
> > > > >
> > > > > I haven't set exact dates for the various stages (feature freeze,
> > code
> > > > > freeze, etc.) for now as Ewen is going to send out an email with
> some
> > > > > suggested tweaks based on his experience as release manager for
> > > 0.10.2.0.
> > > > > We can set the exact dates after that discussion.
> > > > >
> > > > > As we are starting the process early this time, we should expect
> the
> > > > > number
> > > > > of KIPs in the plan to grow (so don't worry if your KIP is not
> there
> > > > yet),
> > > > > but it's good to see that we already have 10 (including 2 merged
> and
> > 2
> > > > > with
> > > > > PR reviews in progress).
> > > > >
> > > > > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > > > KIP-101
> > > > > (Leader Generation in Replication) require message format changes,
> > > which
> > > > > typically imply a major version bump (i.e. 0.11.0.0). If we do
> that,
> > > then
> > > > > it makes sense to also include KIP-106 (Unclean leader election
> > should
> > > be
> > > > > false by default) and KIP-118 (Drop support for Java 7). We would
> > also
> > > > > take
> > > > > the chance to remove deprecated code, in that case.
> > > > >
> > > > > Given the above, how do people feel about 0.11.0.0 as the next
> Kafka
> > > > > version? Please share your thoughts.
> > > > >
> > > > > Thanks,
> > > > > Ismael
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

[Molnár Bálint]

+1

2017-02-28 12:17 GMT+01:00 Dongjin Lee :

>
>
> +1.
>
>
>
> Best,
>
> Dongjin
>
>
>  --
>
>
>
>
> Dongjin Lee
>
>
>
>
>
> Software developer in Line+.
>
> So interested in massive-scale machine learning.
>
>
>
>  facebook:   www.facebook.com/dongjin.lee.kr (http://www.facebook.com/
> dongjin.lee.kr)
>
> linkedin:   kr.linkedin.com/in/dongjinleekr (http://kr.linkedin.com/in/
> dongjinleekr)
>
>
> github:   (http://goog_969573159/)github.com/dongjinleekr (
> http://github.com/dongjinleekr)
>
> twitter:   www.twitter.com/dongjinleekr (http://www.twitter.com/
> dongjinleekr)
>
>
>
>
>
>
>
>
>
>
>
> >
> > On Feb 28, 2017 at 6:40 PM,  mailto:ism...@juma.me.uk)>
> wrote:
> >
> >
> >
> >  Hi everyone,
> >
> > Since the few who responded in the discuss thread were in favour and
> there
> > were no objections, I would like to initiate the voting process for
> > KIP-119: Drop Support for Scala 2.10 in Kafka 0.11:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11
> >
> > The vote will run for a minimum of 72 hours.
> >
> > Thanks,
> > Ismael
> >
>