Hi, I just rebased the Etcd implementation proposal on trunk. Pinging to see if anyone has feedback on my questions from my previous email.
Molnár Bálint <molnarcsi...@gmail.com> ezt írta (időpont: 2018. ápr. 4., Sze, 10:08): > Hi, > Thanks again for the feedback. > > Is there already ongoing work for having an own consensus implementation > within Kafka? > If that work haven't started yet, we think there is value in having an > interim solution, that allows the use of another consensus system besides > Zookeeper. > > We ask the community to take a look at the Etcd implementation proposal > we created and provide feedback on that. > This helps to asses rather this approach is viable at all. > > We are open to collaborate on integrating our proposed Etcd implementation > into any integration test system, to certify that all use cases works as > expected. > > Balint > > 2018-03-30 22:21 GMT+02:00 Gwen Shapira <g...@confluent.io>: > >> Hi, >> >> I had an offline discussion with Ismael and wanted to summarize the >> comments and questions he raised so we are all on the same page. >> >> The core issue is that this change adds a new public API. Since we already >> know that the goal for the next 1-2 years is to get rid of ZK completely. >> Do we want to go to the effort of adding (and discussing and reviewing) a >> new public API, knowing that it will be completely removed in a year? And >> since building and testing a plugin also involves effort, will anyone do >> it >> for something that is going to be temporary by design? >> >> Ismael, correct me if this isn't a fair representation of your concerns. >> >> Gwen >> >> >> >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io> wrote: >> >> > Few other concerns that were raised in the previous discussion were >> around >> > the challenges both to maintainers and users in making this API >> pluggable >> > and how does making the interface pluggable aligns with future goals for >> > the project. At the time this was difficult to discuss because there >> wasn't >> > a concrete proposal. I want to discuss these points in the context of >> this >> > specific proposal: >> > >> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple >> > implementations to cover. >> > In this case: At the time, the Kafka project didn't have much >> > experience with pluggable APIs and components, so the concerns were very >> > valid. Right now Kafka has multiple pluggable components - Connectors, >> > converters, transformations, authentication protocols, authorization >> > database, coordination protocol, serializers, etc. I think that as a >> > community we gotten better at testing the interface, testing the very >> few >> > implementations that are included in Apache Kafka itself and allowing >> the >> > community to innovate and validate outside of the Kafka project. I don't >> > recall major issues either from lack of testing or from usability >> > perspective. >> > >> > 2. Problem: Users don't want to choose a consensus implementation, they >> > just don't want ZK. >> > In this case: I agree that users don't actually want to spend time >> > choosing consensus implementation and a simpler deployment model would >> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK >> > implementation, 99% of the users will choose to use that (a vast >> majority >> > uses our less-than-amazing authorization plugin), and the few that >> really >> > need something else for whatever reason, will be able to get what they >> > need. As Jake said, we need to face the fact that development >> trajectory of >> > ZK isn't amazing at the moment, that it is lacking features our users >> need >> > (SSL) and it will be good to allow the user community to explore >> > alternatives. >> > >> > 3. Problem: Why got to the effort of refactoring if we know we want to >> get >> > rid of ZK. >> > In this case: This change isn't huge, it doesn't rewrite large >> > portions of Kafka and it does not make the future direction any more >> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less >> > solution, applying it on Kafka with this KIP isn't any more challenging >> > than applying it to Kafka without this KIP. It is a step in an >> orthogonal >> > direction, but not opposite direction. I think that letting the perfect >> > become the enemy of the good is a repeated failure mode in this >> community. >> > Can we discuss whether this proposal is good even if there is a more >> > complex proposal that may be better? As long as they don't conflict? >> > >> > Gwen >> > >> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint <molnarcsi...@gmail.com> >> > wrote: >> > >> >> Thanks, for the feedback. >> >> >> >> Developing an internal consensus service inside Kafka would require a >> team >> >> dedicated to this task. >> >> We second what Flavio said in >> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f >> >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org >> %3E >> >> that >> >> getting an implementation which really works and is maintainable is >> >> difficult. >> >> We think that Kafka being able to use another widely used consensus >> system >> >> beside Zookeeper its a safer and workable solution. >> >> It will be easier for users to use a consensus system with Kafka that >> they >> >> are already familiar with. >> >> >> >> >> >> The implementation found here: >> >> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/ >> >> kafka-on-etcd/core/src/main/scala/kafka/etcd >> >> is a first version of enabling Etcd in Kafka. >> >> This implementation hooked in Etcd with a slight change in the existing >> >> interfaces. While this implementation works its far from being >> complete. >> >> Ideally existing interfaces should be reworked to abstract out the used >> >> consensus system. >> >> We opened this KIP to start a discussion and the community to have a >> look >> >> at the initial implementation and receive feedback if this initiative >> is >> >> viable. >> >> >> >> Balint >> >> >> >> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>: >> >> >> >> > I can understand the concerns about the plugability of >> Zookeeper/Etcd. >> >> It >> >> > would not be good for Kafka community if it splits into several >> groups >> >> > using different implementations. >> >> > >> >> > On the other hand, Zookeeper development seems to be a bit stalled. >> So >> >> > maybe there should be at least a discussion whether it makes sense to >> >> > replace Zookeeper with something like Etcd or not. >> >> > >> >> > JAkub >> >> > >> >> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint < >> molnarcsi...@gmail.com> >> >> > wrote: >> >> > >> >> > > Hi all, >> >> > > >> >> > > I have created KIP-273: Kafka to support using ETCD beside >> Zookeeper >> >> > > >> >> > > Here is the link to the KIP: >> >> > > >> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper >> >> > > >> >> > > Looking forward to the discussion. >> >> > > >> >> > > Thanks, >> >> > > Balint >> >> > > >> >> > >> >> >> > >> > >> > >> > -- >> > *Gwen Shapira* >> > Product Manager | Confluent >> > 650.450.2760 <(650)%20450-2760> | @gwenshap >> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog >> > <http://www.confluent.io/blog> >> > >> > >> >> >> -- >> *Gwen Shapira* >> Product Manager | Confluent >> 650.450.2760 | @gwenshap >> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog >> <http://www.confluent.io/blog> >> > >
