Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

On 2018/09/14 21:00:10, Colin McCabe wrote: > On Fri, Sep 14, 2018, at 00:46, lbari...@gmail.com wrote: > > Hello, Colin! > > > > I have read the discussion on KIP-273 and want to raise this question. > > Deploying Kafka to Kubernetes is a crucial problem in my current work. > > And usage

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

On Fri, Sep 14, 2018, at 00:46, lbari...@gmail.com wrote: > Hello, Colin! > > I have read the discussion on KIP-273 and want to raise this question. > Deploying Kafka to Kubernetes is a crucial problem in my current work. > And usage of ZooKeeper makes troubles, because it works unstable in >

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

On 2018/05/09 14:29:17, Molnár Bálint wrote: > Hi, > I just rebased the Etcd implementation proposal on trunk. Pinging to see if > anyone has feedback on my questions from my previous email. > > Molnár Bálint ezt írta (időpont: 2018. ápr. 4., > Sze, 10:08): > > > Hi, > > Thanks again for

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

On 2018/05/18 13:00:52, Molnár Bálint wrote: > Hi Jakub, > > Yes I tried it and as I saw in various forums it works well for Kafka as > well, but because it mimics the Zookeeper API it creates quite many > metadata related value. > Kafka uses only a subset from the Zookeeper API, so those

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

On 2018/05/18 13:00:52, Molnár Bálint wrote: > Hi Jakub, > > Yes I tried it and as I saw in various forums it works well for Kafka as > well, but because it mimics the Zookeeper API it creates quite many > metadata related value. > Kafka uses only a subset from the Zookeeper API, so those

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi Jakub, Yes I tried it and as I saw in various forums it works well for Kafka as well, but because it mimics the Zookeeper API it creates quite many metadata related value. Kafka uses only a subset from the Zookeeper API, so those values can be omitted by implementing Kafka Zookeeper calls in a

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi Balint, Out of curiosity - have you ever tried this project https://github.com/coreos/zetcd which tries to provide Zookeeper API for Etcd? Thanks & Regards Jakub On Wed, Mar 28, 2018 at 6:18 PM Molnár Bálint wrote: > Hi all, > > I have created KIP-273: Kafka to

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi, To be honest, I'm not sure I understand who is the "we" in the "We're going to post ...". If there are any discussions or plans to replace Zookeeper, then I think they should be done here on the developer list instead of teasing people for several months with some secretive projects. Thanks

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi Colin, Do you have a rough estimate on that? Thanks, Balint Colin McCabe ezt írta (időpont: 2018. máj. 9., Sze, 19:53): > Hi Molnar, > > The points Ismael brought up earlier (and that were brought up on KIP-30) > are still relevant here. As Ismael said, the goal is to

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi Molnar, The points Ismael brought up earlier (and that were brought up on KIP-30) are still relevant here. As Ismael said, the goal is to get rid of external dependencies here. We're going to post more about this soon (sorry for the delay) thanks, Colin On Wed, May 9, 2018, at 07:29,

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi, I just rebased the Etcd implementation proposal on trunk. Pinging to see if anyone has feedback on my questions from my previous email. Molnár Bálint ezt írta (időpont: 2018. ápr. 4., Sze, 10:08): > Hi, > Thanks again for the feedback. > > Is there already ongoing

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi, Thanks again for the feedback. Is there already ongoing work for having an own consensus implementation within Kafka? If that work haven't started yet, we think there is value in having an interim solution, that allows the use of another consensus system besides Zookeeper. We ask the

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi, I had an offline discussion with Ismael and wanted to summarize the comments and questions he raised so we are all on the same page. The core issue is that this change adds a new public API. Since we already know that the goal for the next 1-2 years is to get rid of ZK completely. Do we want

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Few other concerns that were raised in the previous discussion were around the challenges both to maintainers and users in making this API pluggable and how does making the interface pluggable aligns with future goals for the project. At the time this was difficult to discuss because there wasn't

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Thanks, for the feedback. Developing an internal consensus service inside Kafka would require a team dedicated to this task. We second what Flavio said in https://lists.apache.org/thread.html/24ae56e073104c4531cf64f7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E that

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

I can understand the concerns about the plugability of Zookeeper/Etcd. It would not be good for Kafka community if it splits into several groups using different implementations. On the other hand, Zookeeper development seems to be a bit stalled. So maybe there should be at least a discussion

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Not sure if this is entirely relevant, but figure folks might be interested: it's early days for us, but we're currently running Kafka+ZK in AWS on Kubernetes StatefulSets with the magic annotation to publish "not ready" addresses (can't recall what it is off the top of my head). It seems stable

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Here's a link to the previous discussion: https://lists.apache.org/thread.html/2bc187040051008452b40b313db06b476c248ef7a5ed7529afe7b118@1448997154@%3Cdev.kafka.apache.org%3E Ismael On Wed, Mar 28, 2018 at 10:40 AM, Ismael Juma wrote: > Hi Gwen, > > I don't think the reasons

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi Gwen, I don't think the reasons why a pluggable metastore is not desirable have changed. My suggestion is that the KIP should try to address the concerns raised previously as part of the proposal. Ismael On Wed, Mar 28, 2018 at 10:24 AM, Gwen Shapira wrote: > While I'm

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

While I'm not in favor of the proposal, I want to point out that the ecosystem changed quite a bit since KIP-30 was first proposed. Kubernetes deployments are far more common now and are growing in popularity, and the problem in deployment, discovery and management that ZK poses is therefore more

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Thanks for the KIP. This was proposed previously via "KIP-30 Allow for brokers to have plug-able consensus and meta data storage sub systems" and the community was not in favour. Have you considered the points discussed then? Ismael On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint