+1 I agree on the governance comments whole heartedly.
Also i agree about the contribution comments made earlier in the thread, i personally am less likely to spend any of mine, or give project time within my internal projects to developers contributing to another commercial companies project even so technically open source, as then there is that commercial companies interest will always prevail and essentially can always have a final vote where disagreement. Im sure they never intend to, but there is that true reality. This is why we have community open source projects. I can find many different implementations now of a rest endpoint on GitHub, BitBucket etc. Each one has their benefits and disadvantages in their implementation. By making / providing one this would bring together these solutions, unifying those developers and also bringing the best of all. I understand the concern on the community burden adding/supporting more surface area for every client. But something like REST is universal and worthy to be owned by the community. Mike ________________________________________ From: Andrew Schofield <andrew_schofield_j...@outlook.com> Sent: Saturday, October 8, 2016 1:19 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-80: Kafka REST Server There's a massive difference between the governance of Kafka and the governance of the REST proxy. In Kafka, there is a broad community of people contributing their opinions about future enhancements in the form of KIPs. There's some really deep consideration that goes into some of the trickier KIPs. There are people outside Confluent deeply knowledgeable in Kafka and building the reputations to become committers. I get the impression that the roadmap of Kafka is not really community-owned (what's the big feature for Kafka 0.11, for example), but the conveyor belt of smaller features in the form of KIPs works nicely. It's a good example of open-source working well. The equivalent for the REST proxy is basically issues on GitHub. The roadmap is less clear. There's not really a community properly engaged in the way that there is with Kafka. So, you could say that it's clear that fewer people are interested, but I think the whole governance thing is a big barrier to engagement. And it's looking like it's getting out of date. In technical terms, I can think of two big improvements to the REST proxy. First, it needs to use the new consumer API so that it's possible to secure connections between the REST proxy and Kafka. I don't care too much which method calls it uses actually uses to consume messages, but I do care that I cannot secure connections because of network security rules. Second, there's an affinity between a consumer and the instance of the REST proxy to which it first connected. Kafka itself avoids this kind of affinity for good reason, and in the name of availability the REST proxy should too. These are natural KIPs. I think it would be good to have the code for the REST proxy contributed to Apache Kafka so that it would be able to be developed in the same way. Andrew Schofield From: Suresh Srinivas <sur...@hortonworks.com> Sent: 07 October 2016 22:41:52 To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-80: Kafka REST Server ASF already gives us a clear framework and governance model for community development. This is already understood by the people contributing to Apache Kafka project, and they are the same people who want to contribute to the REST server capability as well. Everyone is in agreement on the need for collaborating on this effort. So why not contribute the code to Apache Kafka. This will help avoid duplication of effort and forks that may crop up, hugely benefitting the user community. This will also avoid having to define a process similar to ASF on a GitHub project and instead there is a single community with clear understanding community process as defined in ASF. As others have said, this is an important capability for Apache Kafka. It is worth maintaining this as a part of the project. Regards, Suresh On 10/6/16, 8:32 AM, "Ofir Manor" <ofir.ma...@equalum.io> wrote: >I personally think it would be quite wasteful to re-implement the REST >gateway just because that an actively-maintained piece of Apache-licensed >software is not governed directly by the Apache Kafka community... While >kafka-rest repo is owned by Confluent, the contributors including the main >one are also part of the Apache Kafka community, so there is a chance to >work this out. > >However, there are two valid concerns here that could be addressed, around >community and accessibility: >>> What we are worried about is a project >>> that's not maintained in a community. So the process of accepting >>>patches >>> and priorities is not clear, and it's not developed in Apache >>>community. >>> Not only that, existing REST API project doesn't support new client API >and >>> hence there is no security support either. > >This might be easy to fix. Maybe Confluent / kafka-rest community can >clarify that - what is their contribution policy, dev style, roadmap etc. >If they want, they can make an effort to encourage participation from >people outside Confluent (easily accept contributions, invite external >commiters or have open dev process similar to Apache Kafka etc), as there >is definitely seems to be some interest on the list. That might clear the >community concern and help kafka-rest project (but that is a calculation >Confluent will have to make). > >The other, independent, concern is that REST is something that is expected >to be available out of the box with Kafka. I personally don't feel >strongly >about it (better use proper, efficient APIs from day one), though it is >definitely way smaller than adding a stream processing engine to the >project :) >Again,the kafka-rest "community" could take steps to make it even easier >to >install, configure and run kafka-rest for new users on vanilla Apache >Kafka >(outside the Confluent platform), if they wish that (or welcome >contributions to that end), but that is up to them. >Finally, if after the above steps were taken there would still a strong >desire to include a great rest gateway with Apache Kafka, I assume the >community could hypothetically fork the existing kafka-rest into an Apache >Kafka subproject and maintain it "within Apache" instead of implementing >it >from scratch (though I'm not a lawyer etc) - but I cannot imagine it >happen >without Confluent blessing, and I think that is likely much less optimal >(pulling in other Confluent / Apache licensed dependencies) than having a >separate external community around kafka-rest. > > >Just my two cents, > > >Ofir Manor > >Co-Founder & CTO | Equalum > >Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io> >wrote: > >> Neha & Jay, >> We did look at the open source alternatives. Our >>concern >> is what's the patch acceptance and adding features/ bug-fixes to the >> existing project under a Github (although it's licensed under Apache >>2.0). >> It would be great if that project made available under Apache and >>driven by >> the community. Adding to the above, not all Kafka users are interested >>in >> using the Java client API, they would like to have simple REST API where >> they can code against using any language. I do believe this adds value >>to >> Apache Kafka in itself. >> >> "For 1, I don't think there is value in giving in to the NIH syndrome >>and >> reinventing the wheel. What I'm looking for is a detailed comparison of >>the >> gaps and why those can't be improved in the REST proxy that already >>exists >> and is actively maintained." >> >> We are not looking at this as NIH. What we are worried about is a >>project >> that's not maintained in a community. So the process of accepting >>patches >> and priorities is not clear, and it's not developed in Apache community. >> Not only that, existing REST API project doesn't support new client API >>and >> hence there is no security support either. >> We don't know the timeline when that's made available. We would like to >>add >> admin functionality into the REST API. So the Roadmap of that project is >> not driven by Apache. >> >> >> "This doesn't materially have an impact on expanding the usability of >> Kafka. In my experience, REST proxy + Java clients only cover ~50% of >> all >> Kafka users, and maybe 10% of those are the ones who will use the >>REST >> proxy. The remaining 50% are non-java client users (C, python, go, >>node >> etc)." >> >> REST API is most often asked feature in my interactions with Kafka >>users. >> In an organization, There will be independent teams who will write their >> Kafka clients using different language libraries available today, and >> there is no way to standardize this. Instead of supporting several >> different client libraries users will be interested in using a REST API >> server. The need for a REST API will only increase as more and more >>users >> start using Kafka. >> >> "More surface area means more work to keep things consistent. Failure >> to do that has, in fact, hurt the user experience." >> Having myriad Kafka client GitHub projects that support different >>languages >> hurts the user experience and pushes burden to maintain these libraries. >> REST API is a simple code base that uses existing java client libraries >>to >> make life easier for the users. >> >> Thanks, >> Harsha >> >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <n...@confluent.io> wrote: >> >> > Manikumar, >> > >> > Thanks for sharing the proposal. I think there are 2 parts to this >> > discussion - >> > >> > 1. Should we rewrite a REST proxy given that there is a >>feature-complete, >> > open-source and actively maintained REST proxy in the community? >> > 2. Does adding a REST proxy to Apache Kafka make us more agile and >> maintain >> > the high-quality experience that Kafka users have today? >> > >> > For 1, I don't think there is value in giving in to the NIH syndrome >>and >> > reinventing the wheel. What I'm looking for is a detailed comparison >>of >> the >> > gaps and why those can't be improved in the REST proxy that already >> exists >> > and is actively maintained. For example, we depend on zkClient and >>have >> > found as well as fixed several bugs by working closely with the people >> who >> > maintain zkClient. This should be possible for REST proxy as well, >>right? >> > >> > For 2, I'd like us to review our history of expanding the surface >>area to >> > add more clients in the past. Here is a summary - >> > >> > - This doesn't materially have an impact on expanding the >>usability of >> > Kafka. In my experience, REST proxy + Java clients only cover ~50% >>of >> > all >> > Kafka users, and maybe 10% of those are the ones who will use the >>REST >> > proxy. The remaining 50% are non-java client users (C, python, go, >> node >> > etc). >> > - People are a lot more excited about promising to contribute while >> > adding the surface area but not on an ongoing basis down the line. >> > - More surface area means more work to keep things consistent. >>Failure >> > to do that has, in fact, hurt the user experience. >> > - More surface area hurts agility. We want to do a few things >>really >> > well as well as be agile to be able to build on our core >>competency. >> > >> > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <manikumar.re...@gmail.com> >> > wrote: >> > >> > > Hi Jay, >> > > >> > > Thanks for your reply. >> > > >> > > I agree that we can not add all the clients/tools available in >> ecosystem >> > > page to Kafka repo itself. But we feel REST Interface is different >>from >> > > other clients/tools. Since any language that can work with HTTP can >> > > easily integrate with this interface, Having an "official" REST >> > > interface helps user community. This also helps us to integrate well >> > > with external management and provisioning tools. Apache Kafka >>release >> > > with Java clients + REST interface is sufficient for most of the >>user >> > > deployments/requirements. This helps users to deal with less number >> > > of distributions/builds. >> > > >> > > Thanks, >> > > Manikumar >> > > >> > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <j...@confluent.io> wrote: >> > > >> > > > Hey guys, >> > > > >> > > > There's already a REST interface maintained as a separate >> project--it's >> > > > open source and apache licensed and actively maintained ( >> > > > https://github.com/confluentinc/kafka-rest). What is wrong with GitHub - confluentinc/kafka-rest: REST Proxy for Kafka github.com The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and ... >> that? >> > > You >> > > > mentioned that there was some compatibility concern, but >> compatibility >> > > has >> > > > to do with the consumer protocol guarantees not the repo the code >>is >> > in, >> > > > right? Not sure that concern makes sense. >> > > > >> > > > We could argue for adding pretty much anything and everything in >>the >> > > > ecosystem page in Kafka itself but I'm not sure that would make >>the >> > > project >> > > > more agile. >> > > > >> > > > -Jay >> > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar < >> manikumar.re...@gmail.com >> > > >> > > > wrote: >> > > > >> > > > > Hi Kafka Devs, >> > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. >> > > > > >> > > > > There are already open-source alternatives are available. But >>we >> > would >> > > > > like to add REST server that >> > > > > many users ask for under Apache Kafka repo. Many data Infra >>tools >> > comes >> > > > up >> > > > > with Rest Interface. >> > > > > It is useful to have inbuilt Rest API support for Produce, >>Consume >> > > > messages >> > > > > and admin interface for >> > > > > integrating with external management and provisioning tools.This >> will >> > > > also >> > > > > allow the maintenance of >> > > > > REST server and adding new features makes it easy because apache >> > > > community. >> > > > > >> > > > > The KIP wiki is the following: >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > > > > 80%3A+Kafka+Rest+Server >> > > > > >> > > > > Your comments and feedback are welcome. >> > > > > >> > > > > Thanks, >> > > > > Manikumar >> > > > > >> > > > >> > > >> > -- >> > Thanks, >> > Neha >> > >> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
