Hi Colin, Do you have a rough estimate on that?
Thanks, Balint Colin McCabe <cmcc...@apache.org> ezt írta (időpont: 2018. máj. 9., Sze, 19:53): > Hi Molnar, > > The points Ismael brought up earlier (and that were brought up on KIP-30) > are still relevant here. As Ismael said, the goal is to get rid of > external dependencies here. We're going to post more about this soon > (sorry for the delay) > > thanks, > Colin > > > On Wed, May 9, 2018, at 07:29, Molnár Bálint wrote: > > Hi, > > I just rebased the Etcd implementation proposal on trunk. Pinging to see > if > > anyone has feedback on my questions from my previous email. > > > > Molnár Bálint <molnarcsi...@gmail.com> ezt írta (időpont: 2018. ápr. 4., > > Sze, 10:08): > > > > > Hi, > > > Thanks again for the feedback. > > > > > > Is there already ongoing work for having an own consensus > implementation > > > within Kafka? > > > If that work haven't started yet, we think there is value in having an > > > interim solution, that allows the use of another consensus system > besides > > > Zookeeper. > > > > > > We ask the community to take a look at the Etcd implementation proposal > > > we created and provide feedback on that. > > > This helps to asses rather this approach is viable at all. > > > > > > We are open to collaborate on integrating our proposed Etcd > implementation > > > into any integration test system, to certify that all use cases works > as > > > expected. > > > > > > Balint > > > > > > 2018-03-30 22:21 GMT+02:00 Gwen Shapira <g...@confluent.io>: > > > > > >> Hi, > > >> > > >> I had an offline discussion with Ismael and wanted to summarize the > > >> comments and questions he raised so we are all on the same page. > > >> > > >> The core issue is that this change adds a new public API. Since we > already > > >> know that the goal for the next 1-2 years is to get rid of ZK > completely. > > >> Do we want to go to the effort of adding (and discussing and > reviewing) a > > >> new public API, knowing that it will be completely removed in a year? > And > > >> since building and testing a plugin also involves effort, will anyone > do > > >> it > > >> for something that is going to be temporary by design? > > >> > > >> Ismael, correct me if this isn't a fair representation of your > concerns. > > >> > > >> Gwen > > >> > > >> > > >> > > >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io> > wrote: > > >> > > >> > Few other concerns that were raised in the previous discussion were > > >> around > > >> > the challenges both to maintainers and users in making this API > > >> pluggable > > >> > and how does making the interface pluggable aligns with future > goals for > > >> > the project. At the time this was difficult to discuss because there > > >> wasn't > > >> > a concrete proposal. I want to discuss these points in the context > of > > >> this > > >> > specific proposal: > > >> > > > >> > 1. Problem: Pluggable APIs mean larger surface testing area and > multiple > > >> > implementations to cover. > > >> > In this case: At the time, the Kafka project didn't have much > > >> > experience with pluggable APIs and components, so the concerns were > very > > >> > valid. Right now Kafka has multiple pluggable components - > Connectors, > > >> > converters, transformations, authentication protocols, authorization > > >> > database, coordination protocol, serializers, etc. I think that as a > > >> > community we gotten better at testing the interface, testing the > very > > >> few > > >> > implementations that are included in Apache Kafka itself and > allowing > > >> the > > >> > community to innovate and validate outside of the Kafka project. I > don't > > >> > recall major issues either from lack of testing or from usability > > >> > perspective. > > >> > > > >> > 2. Problem: Users don't want to choose a consensus implementation, > they > > >> > just don't want ZK. > > >> > In this case: I agree that users don't actually want to spend > time > > >> > choosing consensus implementation and a simpler deployment model > would > > >> > serve them better. IMO, if Apache Kafka ships with our well-tested > ZK > > >> > implementation, 99% of the users will choose to use that (a vast > > >> majority > > >> > uses our less-than-amazing authorization plugin), and the few that > > >> really > > >> > need something else for whatever reason, will be able to get what > they > > >> > need. As Jake said, we need to face the fact that development > > >> trajectory of > > >> > ZK isn't amazing at the moment, that it is lacking features our > users > > >> need > > >> > (SSL) and it will be good to allow the user community to explore > > >> > alternatives. > > >> > > > >> > 3. Problem: Why got to the effort of refactoring if we know we want > to > > >> get > > >> > rid of ZK. > > >> > In this case: This change isn't huge, it doesn't rewrite large > > >> > portions of Kafka and it does not make the future direction any more > > >> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less > > >> > solution, applying it on Kafka with this KIP isn't any more > challenging > > >> > than applying it to Kafka without this KIP. It is a step in an > > >> orthogonal > > >> > direction, but not opposite direction. I think that letting the > perfect > > >> > become the enemy of the good is a repeated failure mode in this > > >> community. > > >> > Can we discuss whether this proposal is good even if there is a more > > >> > complex proposal that may be better? As long as they don't conflict? > > >> > > > >> > Gwen > > >> > > > >> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint < > molnarcsi...@gmail.com> > > >> > wrote: > > >> > > > >> >> Thanks, for the feedback. > > >> >> > > >> >> Developing an internal consensus service inside Kafka would > require a > > >> team > > >> >> dedicated to this task. > > >> >> We second what Flavio said in > > >> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f > > >> >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@% > 3Cdev.kafka.apache.org > > >> %3E > > >> >> that > > >> >> getting an implementation which really works and is maintainable is > > >> >> difficult. > > >> >> We think that Kafka being able to use another widely used consensus > > >> system > > >> >> beside Zookeeper its a safer and workable solution. > > >> >> It will be easier for users to use a consensus system with Kafka > that > > >> they > > >> >> are already familiar with. > > >> >> > > >> >> > > >> >> The implementation found here: > > >> >> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/ > > >> >> kafka-on-etcd/core/src/main/scala/kafka/etcd > > >> >> is a first version of enabling Etcd in Kafka. > > >> >> This implementation hooked in Etcd with a slight change in the > existing > > >> >> interfaces. While this implementation works its far from being > > >> complete. > > >> >> Ideally existing interfaces should be reworked to abstract out the > used > > >> >> consensus system. > > >> >> We opened this KIP to start a discussion and the community to have > a > > >> look > > >> >> at the initial implementation and receive feedback if this > initiative > > >> is > > >> >> viable. > > >> >> > > >> >> Balint > > >> >> > > >> >> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>: > > >> >> > > >> >> > I can understand the concerns about the plugability of > > >> Zookeeper/Etcd. > > >> >> It > > >> >> > would not be good for Kafka community if it splits into several > > >> groups > > >> >> > using different implementations. > > >> >> > > > >> >> > On the other hand, Zookeeper development seems to be a bit > stalled. > > >> So > > >> >> > maybe there should be at least a discussion whether it makes > sense to > > >> >> > replace Zookeeper with something like Etcd or not. > > >> >> > > > >> >> > JAkub > > >> >> > > > >> >> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint < > > >> molnarcsi...@gmail.com> > > >> >> > wrote: > > >> >> > > > >> >> > > Hi all, > > >> >> > > > > >> >> > > I have created KIP-273: Kafka to support using ETCD beside > > >> Zookeeper > > >> >> > > > > >> >> > > Here is the link to the KIP: > > >> >> > > > > >> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >> >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper > > >> >> > > > > >> >> > > Looking forward to the discussion. > > >> >> > > > > >> >> > > Thanks, > > >> >> > > Balint > > >> >> > > > > >> >> > > > >> >> > > >> > > > >> > > > >> > > > >> > -- > > >> > *Gwen Shapira* > > >> > Product Manager | Confluent > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap > > >> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > >> > <http://www.confluent.io/blog> > > >> > > > >> > > > >> > > >> > > >> -- > > >> *Gwen Shapira* > > >> Product Manager | Confluent > > >> 650.450.2760 | @gwenshap > > >> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > > >> <http://www.confluent.io/blog> > > >> > > > > > > >
