FWIW, Apache Kafka has evolved a lot from where it started. It did start as a messaging system. Over time we realized that that the vision for Kafka is to build a streaming platform and not just a messaging system. You can take a look at the site for more description about what comprises the streaming platform http://kafka.apache.org/ and http://kafka.apache.org/intro.
Can the streaming platform exist without Connect? - No. Data integration is fundamental to building an end to end platform Can the streaming platform exist without stream processing? - No. Processing stream data again is a core part of streaming platform. Can the streaming platform exist without clients? - We at least need one client library to complete the platform. Our Java clients help us to complete the platform story. The rest of the clients are built and maintained outside the project. Can the platform exist without the rest proxy? - Yes. The proxy does not complete the platform vision in anyway. It is just a good to have tool that might be required by quite a few users and there is an active project that works on this - https://github.com/confluentinc/kafka-rest On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis <nso...@linkedin.com.invalid> wrote: > Are you saying Kafka REST is subjective but Kafka Streams and Kafka Connect > are not subjective? > > > "there are likely places that can live without a rest proxy" > > There are also places that can live without Kafka Streams and Kafka > Connect. > > Nacho > > On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <j...@confluent.io> wrote: > > > At the high level, I think ideally it makes sense to add a component to > > Apache Kafka if (1) it's widely needed and (2) it needs tight integration > > with Kafka core. For Kafka Stream, we do expect stream processing will be > > used widely in the future. Implementation wise, Kafka Stream only > supports > > getting data from Kafka and leverages quite a few of the core > > functionalities in Kafka core. For example, it uses customized rebalance > > callback in the consumer and uses the compacted topic heavily. So, having > > Kafka Stream in the same repo makes it easier for testing when those core > > functionalities evolve over time. Kafka Connect is in the same situation. > > > > For rest proxy, whether it's widely used or not is going to be a bit > > subjective. However, there are likely places that can live without a rest > > proxy. The rest proxy is just a proxy for the regular clients and doesn't > > need to be tightly integrated with Kafka core. So, the case for including > > rest proxy in Apache Kafka is probably not as strong as Kafka Stream and > > Kafka Connect. > > > > Thanks, > > > > Jun > > > > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <michael.pea...@ig.com> > > wrote: > > > > > So from my reading essentially the first question needs to answered/and > > > voted on is: > > > > > > Is Apache Kafka Community only about the Core or does the apache > > community > > > also support some subprojects (and just we need some better way to > manage > > > this) > > > > > > If vote for Core only wins, then the following should be removed: > > > Kafka Connect > > > Kafka Stream > > > > > > If vote for Core only loses (aka we will support subprojects) then: > > > We should look to add Kafka Rest > > > > > > And we should look to see how we can manage better govern and manage > > > submodules. > > > > > > A good example which id propose here is how some other communities in > > > Apache do this. > > > > > > Each Module has a Module Management Committee(MMC), this is like almost > > > the PMC but at a per module basis. > > > > > > This MMC should essentially hold the binding votes for that module. > > > The MMC should be made up of a single representative from each > > > organisation (so no single organisation can fully veto the community > it > > > has to a genuine consenus) > > > The MMC requires at least 3 members (so there cant be a tied vote on 2) > > > For a new Module to be added a MMC committee should be sought > > > A new Module is only capable of being added if the above requirements > can > > > be met (e.g. 3 people wishing to step up, from 3 organisations) so that > > > only actively support modules would be added > > > > > > The PMC reviews each module every 6months or Year. If MMC is inactive, > a > > > vote/call to find replacements if raised, if none are forthcoming > > dropping > > > the MMC to less than 3 then the module moves to "the attic" (very much > > like > > > apache attic but a little more aggressively) > > > > > > This way the PMC does not need to micro manage every module > > > We only add modules where some amount of active support and maintenance > > > and use is provided by the community > > > We have an automatic way to retire old or inactive projects. > > > > > > Thoughts? > > > Mike > > > > > > > > > ________________________________________ > > > From: Harsha Ch <harsha...@gmail.com> > > > Sent: Thursday, October 20, 2016 10:26 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > Jay, > > > REST API is something every user is in need of. If the argument > is > > to > > > clone and write your API, this will do a disservice to the users as > they > > > now have to choose one vs. others instead of keeping one API that is > > > supported in Kafka community. > > > > > > "Pre-emptively re-creating another > > > REST layer when it seems like we all quite agree on what needs to be > done > > > and we have an existing code base for HTTP/Kafka access that is heavily > > > used in production seems quite silly." > > > > > > Exactly our point. Why can't we develop this in Apache Kafka > > > community? Instead of us open sourcing another GitHub project and > > creating > > > a divide in users and another version of API. Let's build this in Kafka > > > Community and use the governance model that is proven to provide vendor > > > free user driven consensus features. The argument that is adding this > > REST > > > server to Kafka will affect the agility of the project doesn't mak > sense. > > > > > > It looks like your argument is either we develop all these small tools > or > > > none at all. We as a community need to look at supporting critical > > > tools/API. Instead of dividing this project into individual external > > > communities. We should build this as part of Kafka which best serves > the > > > needs of users. > > > The Streams and Connect projects that were pushed into Kafka > > could > > > have been left in their own Github projects based on your arguments. > What > > > about the REST API is so different that such that it should stay out of > > the > > > Kafka project? From my experience, more users are asking for the REST > > API. > > > > > > Thanks, > > > Harsha > > > > > > > > > > > > > > > > > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote: > > > > > > > I think the questions around governance make sense, I think we should > > > > really clarify that to make the process more clear so it can be fully > > > > inclusive. > > > > > > > > The idea that we should not collaborate on what is there now, though, > > > > because in the future we might disagree about direction does not > really > > > > make sense to me. If in the future we disagree, that is the beauty of > > > open > > > > source, you can always fork off a copy of the code and start an > > > independent > > > > project either in Apache or elsewhere. Pre-emptively re-creating > > another > > > > REST layer when it seems like we all quite agree on what needs to be > > done > > > > and we have an existing code base for HTTP/kafka access that is > heavily > > > > used in production seems quite silly. > > > > > > > > Let me give some background on how I at least think about these > things. > > > > I've participated in open source projects out of LinkedIn via github > as > > > > well as via the ASF. I don't think there is a "right" answer to how > to > > do > > > > these but rather some tradeoffs. We thought about this quite a lot in > > the > > > > context of Kafka based on the experience with the Hadoop ecosystem as > > > well > > > > as from other open source communities. > > > > > > > > There is a rich ecosystem around Kafka. Many of the projects are > quite > > > > small--single clients or tools that do a single thing well--and > almost > > > none > > > > of them are top level apache projects. I don't think trying to force > > each > > > > of these to turn into independent Apache projects is necessarily the > > best > > > > thing for the ecosystem. > > > > > > > > My observation of how this can go wrong is really what I think has > > > happened > > > > to the Hadoop ecosystem. There you see quite a zoo of projects which > > all > > > > drift apart and don't quite work together well. Coordinating even > > simple > > > > changes and standardization across these is exceptionally difficult. > > The > > > > result is a bit of a mess for users--the pieces just don't really > come > > > > together very well. This makes sense for independent infrastructure > > > systems > > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for every > > > > little tool or helper library has lead to a desirable state. I think > > the > > > > mode of operating where the Hadoop vendors spawn off a few new Apache > > > > projects for each new product initiative, especially since often that > > > > project is only valued by that vendor (and the other vendor has a > > > different > > > > competing Apache project) doesn't necessarily do a better job at > > > producing > > > > high quality communities or high quality software. > > > > > > > > These tools/connects/clients/proxies and other integration pieces can > > > take > > > > many forms, but my take of what makes one of these things good is > that > > it > > > > remains simple, does its one thing well, and cleaves as closely as > > > possible > > > > to the conventions for Kafka itself--i.e. doesn't invent new ways of > > > > monitoring, configuring, etc. For the tools we've contributed we've > > tried > > > > really hard to make them consistent with Kafka as well as with each > > other > > > > in how testing, configuration, monitoring, etc works. > > > > > > > > I think what Apache does superbly well is create a community for > > > managing a > > > > large infrastructure layer like Kafka in a vendor independent way. > > What I > > > > think is less successful is attempting to form full and independent > > > apache > > > > communities around very simple single purpose tools, especially if > you > > > hope > > > > for these to come together into a cohesive toolset across multiple > such > > > > tools. Much of what Apache does--create a collective decision making > > > > process for resolving disagreement, help to trademark and protect the > > > marks > > > > of the project, etc just isn't that relevant for simple > single-purpose > > > > tools. > > > > > > > > So my take is there are a couple of options: > > > > > > > > 1. We can try to put all the small tools into the Apache Project. > I > > > > think this is not the right approach as there is simply too many > of > > > > them, > > > > many in different languages, serving different protocols, > > integrating > > > > with > > > > particular systems, and a single community can't effectively > > maintain > > > > them > > > > all. Doing this would significantly slow the progress of the Kafka > > > > project. > > > > As a protocol for messaging, I don't really see a case for > including > > > > REST > > > > but not MQTT or AMQP which are technically much better suited to > > > > messaging > > > > and both are widely used for that. > > > > 2. We can treat ecosystem projects that aren't top level Apache > > > projects > > > > as invalid and try to recreate them all as Apache projects. > > Honestly, > > > > though, if you go to the Kafka ecosystem page virtually none of > the > > > most > > > > popular add-ons to Kafka are Apache projects. The most successful > > > > things in > > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of > > > other > > > > clients, as well as the existing REST layer have succeeded at > > > developing > > > > communities that actively contribute and use these pieces and I > > don't > > > > know > > > > that that is a bad thing unless that community proves to be > > > uninclusive, > > > > unresponsive, or goes in a bad technical direction--and those are > > > > failure > > > > modes that all open source efforts face. > > > > 3. We can do what I think makes the most sense and try to work > with > > > the > > > > projects that exist in the ecosystem and if the project doesn't > > have a > > > > responsive community or wants to go in a different direction fork > or > > > > recreate that work. > > > > > > > > Of course any person can choose whatever of these options they want. > > But > > > > from my point of view, option (3) has been the path of the community > so > > > far > > > > and I think it has been quite successful. > > > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani < > ka...@harsha.io > > > > > > > wrote: > > > > > > > > > Neha, > > > > > "But I haven't seen any community emails or patches being submitted > > by > > > > you > > > > > guys, so I'm wondering why you are concerned about whether the > > > community > > > > is > > > > > open to accepting patches or not." > > > > > > > > > > I think you are talking about contributing patches to this > repository > > > > > right? https://github.com/confluentinc/kafka-rest . All I am > saying > > > > > the guidelines/governance model is not clear on the project and I > > guess > > > > its > > > > > driven by opening a github issue request. Its the repository owned > > by > > > > > confluent and as much I appreciate that the features we mentioned > are > > > in > > > > > the roadmap and welcoming us to contribute to the project. It > doesn't > > > > > gurantee what we want to add in the furture will be in your > roadmap. > > > > > > > > > > Hence the reason having it part of Kafka community will help a lot > as > > > > other > > > > > users can participate in the discussions. We are happy to drive > any > > > > > feature additions through KIPs which gives everyone a chance to > > > > participate > > > > > and add to the discussions. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce < > > michael.pea...@ig.com> > > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > > > > > Also i agree about the contribution comments made earlier in the > > > > thread, > > > > > i > > > > > > personally am less likely to spend any of mine, or give project > > time > > > > > within > > > > > > my internal projects to developers contributing to another > > commercial > > > > > > companies project even so technically open source, as then there > is > > > > that > > > > > > commercial companies interest will always prevail and essentially > > can > > > > > > always have a final vote where disagreement. Im sure they never > > > intend > > > > > to, > > > > > > but there is that true reality. This is why we have community > open > > > > source > > > > > > projects. > > > > > > > > > > > > I can find many different implementations now of a rest endpoint > on > > > > > > GitHub, BitBucket etc. Each one has their benefits and > > disadvantages > > > in > > > > > > their implementation. By making / providing one this would bring > > > > together > > > > > > these solutions, unifying those developers and also bringing the > > best > > > > of > > > > > > all. > > > > > > > > > > > > I understand the concern on the community burden > adding/supporting > > > more > > > > > > surface area for every client. But something like REST is > universal > > > and > > > > > > worthy to be owned by the community. > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > ________________________________________ > > > > > > From: Andrew Schofield <andrew_schofield_j...@outlook.com> > > > > > > Sent: Saturday, October 8, 2016 1:19 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > > > > > There's a massive difference between the governance of Kafka and > > the > > > > > > governance of the REST proxy. > > > > > > > > > > > > In Kafka, there is a broad community of people contributing their > > > > > opinions > > > > > > about future enhancements in the form of KIPs. There's some > really > > > deep > > > > > > consideration that goes into some of the trickier KIPs. There are > > > > people > > > > > > outside Confluent deeply knowledgeable in Kafka and building the > > > > > > reputations to become committers. I get the impression that the > > > roadmap > > > > > of > > > > > > Kafka is not really community-owned (what's the big feature for > > Kafka > > > > > 0.11, > > > > > > for example), but the conveyor belt of smaller features in the > form > > > of > > > > > KIPs > > > > > > works nicely. It's a good example of open-source working well. > > > > > > > > > > > > The equivalent for the REST proxy is basically issues on GitHub. > > The > > > > > > roadmap is less clear. There's not really a community properly > > > engaged > > > > in > > > > > > the way that there is with Kafka. So, you could say that it's > clear > > > > that > > > > > > fewer people are interested, but I think the whole governance > > thing > > > > is a > > > > > > big barrier to engagement. And it's looking like it's getting out > > of > > > > > date. > > > > > > > > > > > > In technical terms, I can think of two big improvements to the > REST > > > > > proxy. > > > > > > First, it needs to use the new consumer API so that it's possible > > to > > > > > secure > > > > > > connections between the REST proxy and Kafka. I don't care too > much > > > > which > > > > > > method calls it uses actually uses to consume messages, but I do > > > care > > > > > that > > > > > > I cannot secure connections because of network security rules. > > > Second, > > > > > > there's an affinity between a consumer and the instance of the > REST > > > > proxy > > > > > > to which it first connected. Kafka itself avoids this kind of > > > affinity > > > > > for > > > > > > good reason, and in the name of availability the REST proxy > should > > > too. > > > > > > These are natural KIPs. > > > > > > > > > > > > I think it would be good to have the code for the REST proxy > > > > contributed > > > > > > to Apache Kafka so that it would be able to be developed in the > > same > > > > way. > > > > > > > > > > > > Andrew Schofield > > > > > > > > > > > > From: Suresh Srinivas <sur...@hortonworks.com> > > > > > > Sent: 07 October 2016 22:41:52 > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > > > > > ASF already gives us a clear framework and governance model for > > > > community > > > > > > development. This is already understood by the people > contributing > > to > > > > > > Apache Kafka project, and they are the same people who want to > > > > contribute > > > > > > to the REST server capability as well. Everyone is in agreement > on > > > the > > > > > > need for collaborating on this effort. So why not contribute the > > code > > > > to > > > > > > Apache Kafka. This will help avoid duplication of effort and > forks > > > that > > > > > > may crop up, hugely benefitting the user community. This will > also > > > > avoid > > > > > > having to define a process similar to ASF on a GitHub project and > > > > instead > > > > > > there is a single community with clear understanding community > > > process > > > > as > > > > > > defined in ASF. > > > > > > > > > > > > As others have said, this is an important capability for Apache > > > Kafka. > > > > It > > > > > > is worth maintaining this as a part of the project. > > > > > > > > > > > > Regards, > > > > > > Suresh > > > > > > > > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <ofir.ma...@equalum.io> wrote: > > > > > > > > > > > > >I personally think it would be quite wasteful to re-implement > the > > > REST > > > > > > >gateway just because that an actively-maintained piece of > > > > > Apache-licensed > > > > > > >software is not governed directly by the Apache Kafka > community... > > > > While > > > > > > >kafka-rest repo is owned by Confluent, the contributors > including > > > the > > > > > main > > > > > > >one are also part of the Apache Kafka community, so there is a > > > chance > > > > > to > > > > > > >work this out. > > > > > > > > > > > > > >However, there are two valid concerns here that could be > > addressed, > > > > > around > > > > > > >community and accessibility: > > > > > > >>> What we are worried about is a project > > > > > > >>> that's not maintained in a community. So the process of > > accepting > > > > > > >>>patches > > > > > > >>> and priorities is not clear, and it's not developed in Apache > > > > > > >>>community. > > > > > > >>> Not only that, existing REST API project doesn't support new > > > client > > > > > API > > > > > > >and > > > > > > >>> hence there is no security support either. > > > > > > > > > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest > community > > > can > > > > > > >clarify that - what is their contribution policy, dev style, > > roadmap > > > > > etc. > > > > > > >If they want, they can make an effort to encourage participation > > > from > > > > > > >people outside Confluent (easily accept contributions, invite > > > external > > > > > > >commiters or have open dev process similar to Apache Kafka etc), > > as > > > > > there > > > > > > >is definitely seems to be some interest on the list. That might > > > clear > > > > > the > > > > > > >community concern and help kafka-rest project (but that is a > > > > calculation > > > > > > >Confluent will have to make). > > > > > > > > > > > > > >The other, independent, concern is that REST is something that > is > > > > > expected > > > > > > >to be available out of the box with Kafka. I personally don't > feel > > > > > > >strongly > > > > > > >about it (better use proper, efficient APIs from day one), > though > > it > > > > is > > > > > > >definitely way smaller than adding a stream processing engine to > > the > > > > > > >project :) > > > > > > >Again,the kafka-rest "community" could take steps to make it > even > > > > easier > > > > > > >to > > > > > > >install, configure and run kafka-rest for new users on vanilla > > > Apache > > > > > > >Kafka > > > > > > >(outside the Confluent platform), if they wish that (or welcome > > > > > > >contributions to that end), but that is up to them. > > > > > > >Finally, if after the above steps were taken there would still a > > > > strong > > > > > > >desire to include a great rest gateway with Apache Kafka, I > assume > > > the > > > > > > >community could hypothetically fork the existing kafka-rest into > > an > > > > > Apache > > > > > > >Kafka subproject and maintain it "within Apache" instead of > > > > implementing > > > > > > >it > > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot > imagine > > it > > > > > > >happen > > > > > > >without Confluent blessing, and I think that is likely much less > > > > optimal > > > > > > >(pulling in other Confluent / Apache licensed dependencies) than > > > > having > > > > > a > > > > > > >separate external community around kafka-rest. > > > > > > > > > > > > > > > > > > > > >Just my two cents, > > > > > > > > > > > > > > > > > > > > >Ofir Manor > > > > > > > > > > > > > >Co-Founder & CTO | Equalum > > > > > > > > > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> > > <+972%2054-780-1286> | > > > > Email: > > > > > > ofir.ma...@equalum.io > > > > > > > > > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani < > > > ka...@harsha.io > > > > > > > > > > > >wrote: > > > > > > > > > > > > > >> Neha & Jay, > > > > > > >> We did look at the open source alternatives. > > Our > > > > > > >>concern > > > > > > >> is what's the patch acceptance and adding features/ bug-fixes > to > > > the > > > > > > >> existing project under a Github (although it's licensed under > > > Apache > > > > > > >>2.0). > > > > > > >> It would be great if that project made available under Apache > > and > > > > > > >>driven by > > > > > > >> the community. Adding to the above, not all Kafka users are > > > > > interested > > > > > > >>in > > > > > > >> using the Java client API, they would like to have simple REST > > API > > > > > where > > > > > > >> they can code against using any language. I do believe this > adds > > > > value > > > > > > >>to > > > > > > >> Apache Kafka in itself. > > > > > > >> > > > > > > >> "For 1, I don't think there is value in giving in to the NIH > > > > syndrome > > > > > > >>and > > > > > > >> reinventing the wheel. What I'm looking for is a detailed > > > comparison > > > > > of > > > > > > >>the > > > > > > >> gaps and why those can't be improved in the REST proxy that > > > already > > > > > > >>exists > > > > > > >> and is actively maintained." > > > > > > >> > > > > > > >> We are not looking at this as NIH. What we are worried about > > is a > > > > > > >>project > > > > > > >> that's not maintained in a community. So the process of > > accepting > > > > > > >>patches > > > > > > >> and priorities is not clear, and it's not developed in Apache > > > > > community. > > > > > > >> Not only that, existing REST API project doesn't support new > > > client > > > > > API > > > > > > >>and > > > > > > >> hence there is no security support either. > > > > > > >> We don't know the timeline when that's made available. We > would > > > like > > > > > to > > > > > > >>add > > > > > > >> admin functionality into the REST API. So the Roadmap of that > > > > project > > > > > is > > > > > > >> not driven by Apache. > > > > > > >> > > > > > > >> > > > > > > >> "This doesn't materially have an impact on expanding the > > usability > > > > of > > > > > > >> Kafka. In my experience, REST proxy + Java clients only > cover > > > > ~50% > > > > > of > > > > > > >> all > > > > > > >> Kafka users, and maybe 10% of those are the ones who will > use > > > the > > > > > > >>REST > > > > > > >> proxy. The remaining 50% are non-java client users (C, > > python, > > > > go, > > > > > > >>node > > > > > > >> etc)." > > > > > > >> > > > > > > >> REST API is most often asked feature in my interactions with > > Kafka > > > > > > >>users. > > > > > > >> In an organization, There will be independent teams who will > > write > > > > > their > > > > > > >> Kafka clients using different language libraries available > > today, > > > > and > > > > > > >> there is no way to standardize this. Instead of supporting > > several > > > > > > >> different client libraries users will be interested in using a > > > REST > > > > > API > > > > > > >> server. The need for a REST API will only increase as more and > > > more > > > > > > >>users > > > > > > >> start using Kafka. > > > > > > >> > > > > > > >> "More surface area means more work to keep things consistent. > > > > Failure > > > > > > >> to do that has, in fact, hurt the user experience." > > > > > > >> Having myriad Kafka client GitHub projects that support > > different > > > > > > >>languages > > > > > > >> hurts the user experience and pushes burden to maintain these > > > > > libraries. > > > > > > >> REST API is a simple code base that uses existing java client > > > > > libraries > > > > > > >>to > > > > > > >> make life easier for the users. > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Harsha > > > > > > >> > > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede < > > n...@confluent.io> > > > > > > wrote: > > > > > > >> > > > > > > >> > Manikumar, > > > > > > >> > > > > > > > >> > Thanks for sharing the proposal. I think there are 2 parts > to > > > this > > > > > > >> > discussion - > > > > > > >> > > > > > > > >> > 1. Should we rewrite a REST proxy given that there is a > > > > > > >>feature-complete, > > > > > > >> > open-source and actively maintained REST proxy in the > > community? > > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more > agile > > > and > > > > > > >> maintain > > > > > > >> > the high-quality experience that Kafka users have today? > > > > > > >> > > > > > > > >> > For 1, I don't think there is value in giving in to the NIH > > > > syndrome > > > > > > >>and > > > > > > >> > reinventing the wheel. What I'm looking for is a detailed > > > > comparison > > > > > > >>of > > > > > > >> the > > > > > > >> > gaps and why those can't be improved in the REST proxy that > > > > already > > > > > > >> exists > > > > > > >> > and is actively maintained. For example, we depend on > zkClient > > > and > > > > > > >>have > > > > > > >> > found as well as fixed several bugs by working closely with > > the > > > > > people > > > > > > >> who > > > > > > >> > maintain zkClient. This should be possible for REST proxy as > > > well, > > > > > > >>right? > > > > > > >> > > > > > > > >> > For 2, I'd like us to review our history of expanding the > > > surface > > > > > > >>area to > > > > > > >> > add more clients in the past. Here is a summary - > > > > > > >> > > > > > > > >> > - This doesn't materially have an impact on expanding the > > > > > > >>usability of > > > > > > >> > Kafka. In my experience, REST proxy + Java clients only > > cover > > > > > ~50% > > > > > > >>of > > > > > > >> > all > > > > > > >> > Kafka users, and maybe 10% of those are the ones who will > > use > > > > the > > > > > > >>REST > > > > > > >> > proxy. The remaining 50% are non-java client users (C, > > > python, > > > > > go, > > > > > > >> node > > > > > > >> > etc). > > > > > > >> > - People are a lot more excited about promising to > > contribute > > > > > while > > > > > > >> > adding the surface area but not on an ongoing basis down > > the > > > > > line. > > > > > > >> > - More surface area means more work to keep things > > > consistent. > > > > > > >>Failure > > > > > > >> > to do that has, in fact, hurt the user experience. > > > > > > >> > - More surface area hurts agility. We want to do a few > > things > > > > > > >>really > > > > > > >> > well as well as be agile to be able to build on our core > > > > > > >>competency. > > > > > > >> > > > > > > > >> > > > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar < > > > > manikumar.re...@gmail.com > > > > > > > > > > > > >> > wrote: > > > > > > >> > > > > > > > >> > > Hi Jay, > > > > > > >> > > > > > > > > >> > > Thanks for your reply. > > > > > > >> > > > > > > > > >> > > I agree that we can not add all the clients/tools > available > > in > > > > > > >> ecosystem > > > > > > >> > > page to Kafka repo itself. But we feel REST Interface is > > > > different > > > > > > >>from > > > > > > >> > > other clients/tools. Since any language that can work with > > > HTTP > > > > > can > > > > > > >> > > easily integrate with this interface, Having an "official" > > > REST > > > > > > >> > > interface helps user community. This also helps us to > > > integrate > > > > > well > > > > > > >> > > with external management and provisioning tools. Apache > > Kafka > > > > > > >>release > > > > > > >> > > with Java clients + REST interface is sufficient for most > of > > > the > > > > > > >>user > > > > > > >> > > deployments/requirements. This helps users to deal with > less > > > > > number > > > > > > >> > > of distributions/builds. > > > > > > >> > > > > > > > > >> > > Thanks, > > > > > > >> > > Manikumar > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps < > j...@confluent.io > > > > > > > > wrote: > > > > > > >> > > > > > > > > >> > > > Hey guys, > > > > > > >> > > > > > > > > > >> > > > There's already a REST interface maintained as a > separate > > > > > > >> project--it's > > > > > > >> > > > open source and apache licensed and actively maintained > ( > > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is > > wrong > > > > with > > > > > > > > > > > > > > > > > > > > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka > > > > > > github.com > > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka > > cluster. > > > > It > > > > > > makes it easy to produce and consume messages, view the state of > > the > > > > > > cluster, and ... > > > > > > > > > > > > >> that? > > > > > > >> > > You > > > > > > >> > > > mentioned that there was some compatibility concern, but > > > > > > >> compatibility > > > > > > >> > > has > > > > > > >> > > > to do with the consumer protocol guarantees not the repo > > the > > > > > code > > > > > > >>is > > > > > > >> > in, > > > > > > >> > > > right? Not sure that concern makes sense. > > > > > > >> > > > > > > > > > >> > > > We could argue for adding pretty much anything and > > > everything > > > > in > > > > > > >>the > > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that > would > > > > make > > > > > > >>the > > > > > > >> > > project > > > > > > >> > > > more agile. > > > > > > >> > > > > > > > > > >> > > > -Jay > > > > > > >> > > > > > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar < > > > > > > >> manikumar.re...@gmail.com > > > > > > >> > > > > > > > > >> > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > Hi Kafka Devs, > > > > > > >> > > > > > > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka > > > > Repository. > > > > > > >> > > > > > > > > > > >> > > > > There are already open-source alternatives are > > available. > > > > But > > > > > > >>we > > > > > > >> > would > > > > > > >> > > > > like to add REST server that > > > > > > >> > > > > many users ask for under Apache Kafka repo. Many data > > > Infra > > > > > > >>tools > > > > > > >> > comes > > > > > > >> > > > up > > > > > > >> > > > > with Rest Interface. > > > > > > >> > > > > It is useful to have inbuilt Rest API support for > > Produce, > > > > > > >>Consume > > > > > > >> > > > messages > > > > > > >> > > > > and admin interface for > > > > > > >> > > > > integrating with external management and provisioning > > > > > tools.This > > > > > > >> will > > > > > > >> > > > also > > > > > > >> > > > > allow the maintenance of > > > > > > >> > > > > REST server and adding new features makes it easy > > because > > > > > apache > > > > > > >> > > > community. > > > > > > >> > > > > > > > > > > >> > > > > The KIP wiki is the following: > > > > > > >> > > > > https://cwiki.apache.org/ > confluence/display/KAFKA/KIP- > > > > > > >> > > > > 80%3A+Kafka+Rest+Server > > > > > > >> > > > > > > > > > > >> > > > > Your comments and feedback are welcome. > > > > > > >> > > > > > > > > > > >> > > > > Thanks, > > > > > > >> > > > > Manikumar > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > -- > > > > > > >> > Thanks, > > > > > > >> > Neha > > > > > > >> > > > > > > > >> > > > > > > The information contained in this email is strictly confidential > > and > > > > for > > > > > > the use of the addressee only, unless otherwise indicated. If you > > are > > > > not > > > > > > the intended recipient, please do not read, copy, use or disclose > > to > > > > > others > > > > > > this message or any attachment. Please also notify the sender by > > > > replying > > > > > > to this email or by telephone (+44(020 7896 0011) and then delete > > the > > > > > email > > > > > > and any copies of it. Opinions, conclusion (etc) that do not > relate > > > to > > > > > the > > > > > > official business of this company shall be understood as neither > > > given > > > > > nor > > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > company > > > > > > registered in England and Wales, company number 04008957) and IG > > > Index > > > > > > Limited (a company registered in England and Wales, company > number > > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > > > Hill, > > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) > > and > > > > IG > > > > > > Index Limited (register number 114059) are authorised and > regulated > > > by > > > > > the > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > The information contained in this email is strictly confidential and > for > > > the use of the addressee only, unless otherwise indicated. If you are > not > > > the intended recipient, please do not read, copy, use or disclose to > > others > > > this message or any attachment. Please also notify the sender by > replying > > > to this email or by telephone (+44(020 7896 0011) and then delete the > > email > > > and any copies of it. Opinions, conclusion (etc) that do not relate to > > the > > > official business of this company shall be understood as neither given > > nor > > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > > registered in England and Wales, company number 04008957) and IG Index > > > Limited (a company registered in England and Wales, company number > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and > IG > > > Index Limited (register number 114059) are authorised and regulated by > > the > > > Financial Conduct Authority. > > > > > > > > > -- > Nacho (Ignacio) Solis > Kafka > nso...@linkedin.com >
