Hi, I had an offline discussion with Ismael and wanted to summarize the comments and questions he raised so we are all on the same page.
The core issue is that this change adds a new public API. Since we already know that the goal for the next 1-2 years is to get rid of ZK completely. Do we want to go to the effort of adding (and discussing and reviewing) a new public API, knowing that it will be completely removed in a year? And since building and testing a plugin also involves effort, will anyone do it for something that is going to be temporary by design? Ismael, correct me if this isn't a fair representation of your concerns. Gwen On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io> wrote: > Few other concerns that were raised in the previous discussion were around > the challenges both to maintainers and users in making this API pluggable > and how does making the interface pluggable aligns with future goals for > the project. At the time this was difficult to discuss because there wasn't > a concrete proposal. I want to discuss these points in the context of this > specific proposal: > > 1. Problem: Pluggable APIs mean larger surface testing area and multiple > implementations to cover. > In this case: At the time, the Kafka project didn't have much > experience with pluggable APIs and components, so the concerns were very > valid. Right now Kafka has multiple pluggable components - Connectors, > converters, transformations, authentication protocols, authorization > database, coordination protocol, serializers, etc. I think that as a > community we gotten better at testing the interface, testing the very few > implementations that are included in Apache Kafka itself and allowing the > community to innovate and validate outside of the Kafka project. I don't > recall major issues either from lack of testing or from usability > perspective. > > 2. Problem: Users don't want to choose a consensus implementation, they > just don't want ZK. > In this case: I agree that users don't actually want to spend time > choosing consensus implementation and a simpler deployment model would > serve them better. IMO, if Apache Kafka ships with our well-tested ZK > implementation, 99% of the users will choose to use that (a vast majority > uses our less-than-amazing authorization plugin), and the few that really > need something else for whatever reason, will be able to get what they > need. As Jake said, we need to face the fact that development trajectory of > ZK isn't amazing at the moment, that it is lacking features our users need > (SSL) and it will be good to allow the user community to explore > alternatives. > > 3. Problem: Why got to the effort of refactoring if we know we want to get > rid of ZK. > In this case: This change isn't huge, it doesn't rewrite large > portions of Kafka and it does not make the future direction any more > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less > solution, applying it on Kafka with this KIP isn't any more challenging > than applying it to Kafka without this KIP. It is a step in an orthogonal > direction, but not opposite direction. I think that letting the perfect > become the enemy of the good is a repeated failure mode in this community. > Can we discuss whether this proposal is good even if there is a more > complex proposal that may be better? As long as they don't conflict? > > Gwen > > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint <molnarcsi...@gmail.com> > wrote: > >> Thanks, for the feedback. >> >> Developing an internal consensus service inside Kafka would require a team >> dedicated to this task. >> We second what Flavio said in >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E >> that >> getting an implementation which really works and is maintainable is >> difficult. >> We think that Kafka being able to use another widely used consensus system >> beside Zookeeper its a safer and workable solution. >> It will be easier for users to use a consensus system with Kafka that they >> are already familiar with. >> >> >> The implementation found here: >> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/ >> kafka-on-etcd/core/src/main/scala/kafka/etcd >> is a first version of enabling Etcd in Kafka. >> This implementation hooked in Etcd with a slight change in the existing >> interfaces. While this implementation works its far from being complete. >> Ideally existing interfaces should be reworked to abstract out the used >> consensus system. >> We opened this KIP to start a discussion and the community to have a look >> at the initial implementation and receive feedback if this initiative is >> viable. >> >> Balint >> >> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>: >> >> > I can understand the concerns about the plugability of Zookeeper/Etcd. >> It >> > would not be good for Kafka community if it splits into several groups >> > using different implementations. >> > >> > On the other hand, Zookeeper development seems to be a bit stalled. So >> > maybe there should be at least a discussion whether it makes sense to >> > replace Zookeeper with something like Etcd or not. >> > >> > JAkub >> > >> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint <molnarcsi...@gmail.com> >> > wrote: >> > >> > > Hi all, >> > > >> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper >> > > >> > > Here is the link to the KIP: >> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper >> > > >> > > Looking forward to the discussion. >> > > >> > > Thanks, >> > > Balint >> > > >> > >> > > > > -- > *Gwen Shapira* > Product Manager | Confluent > 650.450.2760 <(650)%20450-2760> | @gwenshap > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > <http://www.confluent.io/blog> > > -- *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter <https://twitter.com/ConfluentInc> | blog <http://www.confluent.io/blog>
