Re: [DISCUSS] KIP-80: Kafka REST Server

[Manikumar]

ed to be developed within Kafka.
> > > > > > > >
> > > > > > > >
> > > > > > > >"Can the platform exist without the rest proxy? - Yes. The
> proxy
> > > > does
> > > > > > not
> > > > > > > >complete the platform vision in anyway. It is just a good to
> > have
> > > > tool
> > > > > > > >that
> > > > > > > >might be required by quite a few users and there is an active
> > > > project
> > > > > > that
> > > > > > > >works on this - https://github.com/confluentinc/kafka-rest;
> > > > > > > >
> > > > > > > >The rest proxy is as important as any API. The vision that
> shown
> > > > here
> > > > > > > >http://kafka.apache.org/intro
> > > > > > > >require users to write the producers and consumers to get
> their
> > > data
> > > > > > into
> > > > > > > >and out of Kafka, without which having Streams or Connect
> won't
> > > help
> > > > > > > >anyone.
> > > > > > > >The rest proxy makes easier for users get their data into
> Kafka.
> > > > > > > >Adding the rest proxy to the project doesn't invalidate the
> > > current
> > > > > > > >vision,
> > > > > > > >it only strengthens it.
> > > > > > > >
> > > > > > > >Thanks,
> > > > > > > >Harsha
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <
> > > > r...@confluent.io>
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >FWIW, Apache Kafka has evolved a lot from where it started. It
> > did
> > > > > start
> > > > > > > >as
> > > > > > > >a messaging system. Over time we realized that that the vision
> > for
> > > > > Kafka
> > > > > > > >is
> > > > > > > >to build a streaming platform and not just a messaging system.
> > You
> > > > can
> > > > > > > >take
> > > > > > > >a look at the site for more description about what comprises
> the
> > > > > > streaming
> > > > > > > >platform http://kafka.apache.org/ and
> > > http://kafka.apache.org/intro
> > > > .
> > > > > > > >
> > > > > > > >Can the streaming platform exist without Connect? - No. Data
> > > > > integration
> > > > > > > >is
> > > > > > > >fundamental to building an end to end platform
> > > > > > > >
> > > > > > > >Can the streaming platform exist without stream processing? -
> > No.
> > > > > > > >Processing stream data again is a core part of streaming
> > platform.
> > > > > > > >
> > > > > > > >Can the streaming platform exist without clients? - We at
> least
> > > need
> > > > > one
> > > > > > > >client library to complete the platform. Our Java clients help
> > us
> > > to
> > > > > > > >complete the platform story. The rest of the clients are built
> > and
> > > > > > > >maintained outside the project.
> > > > > > > >
> > > > > > > >Can the platform exist without the rest proxy? - Yes. The
> proxy
> > > does
> > > > > not
> > > > > > > >complete the platform vision in anyway. It is just a good to
> > have
> > > > tool
> > > > > > > >that
> > > > > > > >might be required by quite a few users and there is an active
> > > > project
> > > > > > that
> > > > > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >On Fri, Oct 21, 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Guozhang Wang]

> > > > > > >complete the platform story. The rest of the clients are built
> and
> > > > > > >maintained outside the project.
> > > > > > >
> > > > > > >Can the platform exist without the rest proxy? - Yes. The proxy
> > does
> > > > not
> > > > > > >complete the platform vision in anyway. It is just a good to
> have
> > > tool
> > > > > > >that
> > > > > > >might be required by quite a few users and there is an active
> > > project
> > > > > that
> > > > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > > > ><nso...@linkedin.com.invalid>
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and
> > > Kafka
> > > > > > >Connect
> > > > > > >> are not subjective?
> > > > > > >>
> > > > > > >> > "there are likely places that can live without a rest proxy"
> > > > > > >>
> > > > > > >> There are also places that can live without Kafka Streams and
> > > Kafka
> > > > > > >> Connect.
> > > > > > >>
> > > > > > >> Nacho
> > > > > > >>
> > > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <j...@confluent.io>
> > > wrote:
> > > > > > >>
> > > > > > >> > At the high level, I think ideally it makes sense to add a
> > > > component
> > > > > > >>to
> > > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs
> tight
> > > > > > >integration
> > > > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> > > processing
> > > > > will
> > > > > > >be
> > > > > > >> > used widely in the future. Implementation wise, Kafka Stream
> > > only
> > > > > > >> supports
> > > > > > >> > getting data from Kafka and leverages quite a few of the
> core
> > > > > > >> > functionalities in Kafka core. For example, it uses
> customized
> > > > > > >>rebalance
> > > > > > >> > callback in the consumer and uses the compacted topic
> heavily.
> > > So,
> > > > > > >having
> > > > > > >> > Kafka Stream in the same repo makes it easier for testing
> when
> > > > those
> > > > > > >core
> > > > > > >> > functionalities evolve over time. Kafka Connect is in the
> same
> > > > > > >situation.
> > > > > > >> >
> > > > > > >> > For rest proxy, whether it's widely used or not is going to
> > be a
> > > > bit
> > > > > > >> > subjective. However, there are likely places that can live
> > > > without a
> > > > > > >rest
> > > > > > >> > proxy. The rest proxy is just a proxy for the regular
> clients
> > > and
> > > > > > >doesn't
> > > > > > >> > need to be tightly integrated with Kafka core. So, the case
> > for
> > > > > > >including
> > > > > > >> > rest proxy in Apache Kafka is probably not as strong as
> Kafka
> > > > Stream
> > > > > > >>and
> > > > > > >> > Kafka Connect.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> >
> > > > > > >> > Jun
> > > > > > >> >
> > > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > > > >><michael.pea...@ig.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > So from my reading essentially the first question needs to
> > > >

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ewen Cheslack-Postava]

t;complete the platform story. The rest of the clients are built and
> > > > > >maintained outside the project.
> > > > > >
> > > > > >Can the platform exist without the rest proxy? - Yes. The proxy
> does
> > > not
> > > > > >complete the platform vision in anyway. It is just a good to have
> > tool
> > > > > >that
> > > > > >might be required by quite a few users and there is an active
> > project
> > > > that
> > > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > > ><nso...@linkedin.com.invalid>
> > > > > >wrote:
> > > > > >
> > > > > >> Are you saying Kafka REST is subjective but Kafka Streams and
> > Kafka
> > > > > >Connect
> > > > > >> are not subjective?
> > > > > >>
> > > > > >> > "there are likely places that can live without a rest proxy"
> > > > > >>
> > > > > >> There are also places that can live without Kafka Streams and
> > Kafka
> > > > > >> Connect.
> > > > > >>
> > > > > >> Nacho
> > > > > >>
> > > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <j...@confluent.io>
> > wrote:
> > > > > >>
> > > > > >> > At the high level, I think ideally it makes sense to add a
> > > component
> > > > > >>to
> > > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > > > > >integration
> > > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> > processing
> > > > will
> > > > > >be
> > > > > >> > used widely in the future. Implementation wise, Kafka Stream
> > only
> > > > > >> supports
> > > > > >> > getting data from Kafka and leverages quite a few of the core
> > > > > >> > functionalities in Kafka core. For example, it uses customized
> > > > > >>rebalance
> > > > > >> > callback in the consumer and uses the compacted topic heavily.
> > So,
> > > > > >having
> > > > > >> > Kafka Stream in the same repo makes it easier for testing when
> > > those
> > > > > >core
> > > > > >> > functionalities evolve over time. Kafka Connect is in the same
> > > > > >situation.
> > > > > >> >
> > > > > >> > For rest proxy, whether it's widely used or not is going to
> be a
> > > bit
> > > > > >> > subjective. However, there are likely places that can live
> > > without a
> > > > > >rest
> > > > > >> > proxy. The rest proxy is just a proxy for the regular clients
> > and
> > > > > >doesn't
> > > > > >> > need to be tightly integrated with Kafka core. So, the case
> for
> > > > > >including
> > > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka
> > > Stream
> > > > > >>and
> > > > > >> > Kafka Connect.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > Jun
> > > > > >> >
> > > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > > >><michael.pea...@ig.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > So from my reading essentially the first question needs to
> > > > > >answered/and
> > > > > >> > > voted on is:
> > > > > >> > >
> > > > > >> > > Is Apache Kafka Community only about the Core or does the
> > apache
> > > > > >> > community
> > > > > >> > > also support some subprojects (and just we need some better
> > way
> > > to
> > > > > >> manage
> > > >

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ben Davison]

gt; >Can the streaming platform exist without Connect? - No. Data
> > integration
> > > > >is
> > > > >fundamental to building an end to end platform
> > > > >
> > > > >Can the streaming platform exist without stream processing? - No.
> > > > >Processing stream data again is a core part of streaming platform.
> > > > >
> > > > >Can the streaming platform exist without clients? - We at least need
> > one
> > > > >client library to complete the platform. Our Java clients help us to
> > > > >complete the platform story. The rest of the clients are built and
> > > > >maintained outside the project.
> > > > >
> > > > >Can the platform exist without the rest proxy? - Yes. The proxy does
> > not
> > > > >complete the platform vision in anyway. It is just a good to have
> tool
> > > > >that
> > > > >might be required by quite a few users and there is an active
> project
> > > that
> > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > ><nso...@linkedin.com.invalid>
> > > > >wrote:
> > > > >
> > > > >> Are you saying Kafka REST is subjective but Kafka Streams and
> Kafka
> > > > >Connect
> > > > >> are not subjective?
> > > > >>
> > > > >> > "there are likely places that can live without a rest proxy"
> > > > >>
> > > > >> There are also places that can live without Kafka Streams and
> Kafka
> > > > >> Connect.
> > > > >>
> > > > >> Nacho
> > > > >>
> > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <j...@confluent.io>
> wrote:
> > > > >>
> > > > >> > At the high level, I think ideally it makes sense to add a
> > component
> > > > >>to
> > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > > > >integration
> > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> processing
> > > will
> > > > >be
> > > > >> > used widely in the future. Implementation wise, Kafka Stream
> only
> > > > >> supports
> > > > >> > getting data from Kafka and leverages quite a few of the core
> > > > >> > functionalities in Kafka core. For example, it uses customized
> > > > >>rebalance
> > > > >> > callback in the consumer and uses the compacted topic heavily.
> So,
> > > > >having
> > > > >> > Kafka Stream in the same repo makes it easier for testing when
> > those
> > > > >core
> > > > >> > functionalities evolve over time. Kafka Connect is in the same
> > > > >situation.
> > > > >> >
> > > > >> > For rest proxy, whether it's widely used or not is going to be a
> > bit
> > > > >> > subjective. However, there are likely places that can live
> > without a
> > > > >rest
> > > > >> > proxy. The rest proxy is just a proxy for the regular clients
> and
> > > > >doesn't
> > > > >> > need to be tightly integrated with Kafka core. So, the case for
> > > > >including
> > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka
> > Stream
> > > > >>and
> > > > >> > Kafka Connect.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Jun
> > > > >> >
> > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > >><michael.pea...@ig.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > So from my reading essentially the first question needs to
> > > > >answered/and
> > > > >> > > voted on is:
> > > > >> > >
> > > > >> > > Is Apache Kafka Community only about the Core or does the
> apache
> > > > >> > community
> > > > >&

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ismael Juma]

> > > >
> > > >
> > > >
> > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > ><nso...@linkedin.com.invalid>
> > > >wrote:
> > > >
> > > >> Are you saying Kafka REST is subjective but Kafka Streams and Kafka
> > > >Connect
> > > >> are not subjective?
> > > >>
> > > >> > "there are likely places that can live without a rest proxy"
> > > >>
> > > >> There are also places that can live without Kafka Streams and Kafka
> > > >> Connect.
> > > >>
> > > >> Nacho
> > > >>
> > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <j...@confluent.io> wrote:
> > > >>
> > > >> > At the high level, I think ideally it makes sense to add a
> component
> > > >>to
> > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > > >integration
> > > >> > with Kafka core. For Kafka Stream, we do expect stream processing
> > will
> > > >be
> > > >> > used widely in the future. Implementation wise, Kafka Stream only
> > > >> supports
> > > >> > getting data from Kafka and leverages quite a few of the core
> > > >> > functionalities in Kafka core. For example, it uses customized
> > > >>rebalance
> > > >> > callback in the consumer and uses the compacted topic heavily. So,
> > > >having
> > > >> > Kafka Stream in the same repo makes it easier for testing when
> those
> > > >core
> > > >> > functionalities evolve over time. Kafka Connect is in the same
> > > >situation.
> > > >> >
> > > >> > For rest proxy, whether it's widely used or not is going to be a
> bit
> > > >> > subjective. However, there are likely places that can live
> without a
> > > >rest
> > > >> > proxy. The rest proxy is just a proxy for the regular clients and
> > > >doesn't
> > > >> > need to be tightly integrated with Kafka core. So, the case for
> > > >including
> > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka
> Stream
> > > >>and
> > > >> > Kafka Connect.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jun
> > > >> >
> > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > >><michael.pea...@ig.com>
> > > >> > wrote:
> > > >> >
> > > >> > > So from my reading essentially the first question needs to
> > > >answered/and
> > > >> > > voted on is:
> > > >> > >
> > > >> > > Is Apache Kafka Community only about the Core or does the apache
> > > >> > community
> > > >> > > also support some subprojects (and just we need some better way
> to
> > > >> manage
> > > >> > > this)
> > > >> > >
> > > >> > > If vote for Core only wins, then the following should be
> removed:
> > > >> > > Kafka Connect
> > > >> > > Kafka Stream
> > > >> > >
> > > >> > > If vote for Core only loses (aka we will support subprojects)
> > then:
> > > >> > > We should look to add Kafka Rest
> > > >> > >
> > > >> > > And we should look to see how we can manage better govern and
> > manage
> > > >> > > submodules.
> > > >> > >
> > > >> > > A good example which id propose here is how some other
> communities
> > > >>in
> > > >> > > Apache do this.
> > > >> > >
> > > >> > > Each Module has a Module Management Committee(MMC), this is like
> > > >almost
> > > >> > > the PMC but at a per module basis.
> > > >> > >
> > > >> > > This MMC should essentially hold the binding votes for that
> > module.
> > > >> > > The MMC should be made up of a single representative from each
> > > >> > > organisation  (so no single organisation can fully veto the
> > > >>community
> > > >> it
> > > &g

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ben Davison]

gt;> > getting data from Kafka and leverages quite a few of the core
> > >> > functionalities in Kafka core. For example, it uses customized
> > >>rebalance
> > >> > callback in the consumer and uses the compacted topic heavily. So,
> > >having
> > >> > Kafka Stream in the same repo makes it easier for testing when those
> > >core
> > >> > functionalities evolve over time. Kafka Connect is in the same
> > >situation.
> > >> >
> > >> > For rest proxy, whether it's widely used or not is going to be a bit
> > >> > subjective. However, there are likely places that can live without a
> > >rest
> > >> > proxy. The rest proxy is just a proxy for the regular clients and
> > >doesn't
> > >> > need to be tightly integrated with Kafka core. So, the case for
> > >including
> > >> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream
> > >>and
> > >> > Kafka Connect.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > >><michael.pea...@ig.com>
> > >> > wrote:
> > >> >
> > >> > > So from my reading essentially the first question needs to
> > >answered/and
> > >> > > voted on is:
> > >> > >
> > >> > > Is Apache Kafka Community only about the Core or does the apache
> > >> > community
> > >> > > also support some subprojects (and just we need some better way to
> > >> manage
> > >> > > this)
> > >> > >
> > >> > > If vote for Core only wins, then the following should be removed:
> > >> > > Kafka Connect
> > >> > > Kafka Stream
> > >> > >
> > >> > > If vote for Core only loses (aka we will support subprojects)
> then:
> > >> > > We should look to add Kafka Rest
> > >> > >
> > >> > > And we should look to see how we can manage better govern and
> manage
> > >> > > submodules.
> > >> > >
> > >> > > A good example which id propose here is how some other communities
> > >>in
> > >> > > Apache do this.
> > >> > >
> > >> > > Each Module has a Module Management Committee(MMC), this is like
> > >almost
> > >> > > the PMC but at a per module basis.
> > >> > >
> > >> > > This MMC should essentially hold the binding votes for that
> module.
> > >> > > The MMC should be made up of a single representative from each
> > >> > > organisation  (so no single organisation can fully veto the
> > >>community
> > >> it
> > >> > > has to a genuine consenus)
> > >> > > The MMC requires at least 3 members (so there cant be a tied vote
> on
> > >2)
> > >> > > For a new Module to be added a MMC committee should be sought
> > >> > > A new Module is only capable of being added if the above
> > >>requirements
> > >> can
> > >> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
> > >that
> > >> > > only actively support modules would be added
> > >> > >
> > >> > > The PMC reviews each module every 6months or Year. If MMC is
> > >>inactive,
> > >> a
> > >> > > vote/call to find replacements if raised, if none are forthcoming
> > >> > dropping
> > >> > > the MMC to less than 3 then the module moves to "the attic" (very
> > >>much
> > >> > like
> > >> > > apache attic but a little more aggressively)
> > >> > >
> > >> > > This way the PMC does not need to micro manage every module
> > >> > > We only add modules where some amount of active support and
> > >maintenance
> > >> > > and use is provided by the community
> > >> > > We have an automatic way to retire old or inactive projects.
> > >> > >
> > >> > > Thoughts?
> > >> > > Mike
> > >> > >
> > >> > >
> > >> > > 
> > >&g

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

gt; >> > > voted on is:
> >> > >
> >> > > Is Apache Kafka Community only about the Core or does the apache
> >> > community
> >> > > also support some subprojects (and just we need some better way to
> >> manage
> >> > > this)
> >> > >
> >> > > If vote for Core only wins, then the following should be removed:
> >> > > Kafka Connect
> >> > > Kafka Stream
> >> > >
> >> > > If vote for Core only loses (aka we will support subprojects) then:
> >> > > We should look to add Kafka Rest
> >> > >
> >> > > And we should look to see how we can manage better govern and manage
> >> > > submodules.
> >> > >
> >> > > A good example which id propose here is how some other communities
> >>in
> >> > > Apache do this.
> >> > >
> >> > > Each Module has a Module Management Committee(MMC), this is like
> >almost
> >> > > the PMC but at a per module basis.
> >> > >
> >> > > This MMC should essentially hold the binding votes for that module.
> >> > > The MMC should be made up of a single representative from each
> >> > > organisation  (so no single organisation can fully veto the
> >>community
> >> it
> >> > > has to a genuine consenus)
> >> > > The MMC requires at least 3 members (so there cant be a tied vote on
> >2)
> >> > > For a new Module to be added a MMC committee should be sought
> >> > > A new Module is only capable of being added if the above
> >>requirements
> >> can
> >> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
> >that
> >> > > only actively support modules would be added
> >> > >
> >> > > The PMC reviews each module every 6months or Year. If MMC is
> >>inactive,
> >> a
> >> > > vote/call to find replacements if raised, if none are forthcoming
> >> > dropping
> >> > > the MMC to less than 3 then the module moves to "the attic" (very
> >>much
> >> > like
> >> > > apache attic but a little more aggressively)
> >> > >
> >> > > This way the PMC does not need to micro manage every module
> >> > > We only add modules where some amount of active support and
> >maintenance
> >> > > and use is provided by the community
> >> > > We have an automatic way to retire old or inactive projects.
> >> > >
> >> > > Thoughts?
> >> > > Mike
> >> > >
> >> > >
> >> > > 
> >> > > From: Harsha Ch <harsha...@gmail.com>
> >> > > Sent: Thursday, October 20, 2016 10:26 PM
> >> > > To: dev@kafka.apache.org
> >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >> > >
> >> > > Jay,
> >> > >   REST API is something every user is in need of. If the
> >>argument
> >> is
> >> > to
> >> > > clone and write your  API, this will do a disservice to the users as
> >> they
> >> > > now have to choose one vs. others instead of keeping one API that is
> >> > > supported in Kafka community.
> >> > >
> >> > > "Pre-emptively re-creating another
> >> > > REST layer when it seems like we all quite agree on what needs to be
> >> done
> >> > > and we have an existing code base for HTTP/Kafka access that is
> >heavily
> >> > > used in production seems quite silly."
> >> > >
> >> > >Exactly our point. Why can't we develop this in Apache Kafka
> >> > > community? Instead of us open sourcing another GitHub project and
> >> > creating
> >> > > a divide in users and another version of API. Let's build this in
> >Kafka
> >> > > Community and use the governance model that is proven to provide
> >vendor
> >> > > free user driven consensus features. The argument that is adding
> >>this
> >> > REST
> >> > > server to Kafka will affect the agility of the project doesn't mak
> >> sense.
> >> > >
> >> > > It looks like your argument is either we develop all these small
> >>tools

Re: [DISCUSS] KIP-80: Kafka REST Server

[Suresh Srinivas]

 it's widely needed and (2) it needs tight
>integration
>> > with Kafka core. For Kafka Stream, we do expect stream processing will
>be
>> > used widely in the future. Implementation wise, Kafka Stream only
>> supports
>> > getting data from Kafka and leverages quite a few of the core
>> > functionalities in Kafka core. For example, it uses customized
>>rebalance
>> > callback in the consumer and uses the compacted topic heavily. So,
>having
>> > Kafka Stream in the same repo makes it easier for testing when those
>core
>> > functionalities evolve over time. Kafka Connect is in the same
>situation.
>> >
>> > For rest proxy, whether it's widely used or not is going to be a bit
>> > subjective. However, there are likely places that can live without a
>rest
>> > proxy. The rest proxy is just a proxy for the regular clients and
>doesn't
>> > need to be tightly integrated with Kafka core. So, the case for
>including
>> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream
>>and
>> > Kafka Connect.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
>><michael.pea...@ig.com>
>> > wrote:
>> >
>> > > So from my reading essentially the first question needs to
>answered/and
>> > > voted on is:
>> > >
>> > > Is Apache Kafka Community only about the Core or does the apache
>> > community
>> > > also support some subprojects (and just we need some better way to
>> manage
>> > > this)
>> > >
>> > > If vote for Core only wins, then the following should be removed:
>> > > Kafka Connect
>> > > Kafka Stream
>> > >
>> > > If vote for Core only loses (aka we will support subprojects) then:
>> > > We should look to add Kafka Rest
>> > >
>> > > And we should look to see how we can manage better govern and manage
>> > > submodules.
>> > >
>> > > A good example which id propose here is how some other communities
>>in
>> > > Apache do this.
>> > >
>> > > Each Module has a Module Management Committee(MMC), this is like
>almost
>> > > the PMC but at a per module basis.
>> > >
>> > > This MMC should essentially hold the binding votes for that module.
>> > > The MMC should be made up of a single representative from each
>> > > organisation  (so no single organisation can fully veto the
>>community
>> it
>> > > has to a genuine consenus)
>> > > The MMC requires at least 3 members (so there cant be a tied vote on
>2)
>> > > For a new Module to be added a MMC committee should be sought
>> > > A new Module is only capable of being added if the above
>>requirements
>> can
>> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
>that
>> > > only actively support modules would be added
>> > >
>> > > The PMC reviews each module every 6months or Year. If MMC is
>>inactive,
>> a
>> > > vote/call to find replacements if raised, if none are forthcoming
>> > dropping
>> > > the MMC to less than 3 then the module moves to "the attic" (very
>>much
>> > like
>> > > apache attic but a little more aggressively)
>> > >
>> > > This way the PMC does not need to micro manage every module
>> > > We only add modules where some amount of active support and
>maintenance
>> > > and use is provided by the community
>> > > We have an automatic way to retire old or inactive projects.
>> > >
>> > > Thoughts?
>> > > Mike
>> > >
>> > >
>> > > 
>> > > From: Harsha Ch <harsha...@gmail.com>
>> > > Sent: Thursday, October 20, 2016 10:26 PM
>> > > To: dev@kafka.apache.org
>> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>> > >
>> > > Jay,
>> > >   REST API is something every user is in need of. If the
>>argument
>> is
>> > to
>> > > clone and write your  API, this will do a disservice to the users as
>> they
>> > > now have to choose one vs. others instead of keeping one API that is
>> > > supported in Kafka community.
>> > >
>> > > &quo

Re: [DISCUSS] KIP-80: Kafka REST Server

[Nacho Solis]

t;
> > > wrote:
> > >
> > > > So from my reading essentially the first question needs to
> answered/and
> > > > voted on is:
> > > >
> > > > Is Apache Kafka Community only about the Core or does the apache
> > > community
> > > > also support some subprojects (and just we need some better way to
> > manage
> > > > this)
> > > >
> > > > If vote for Core only wins, then the following should be removed:
> > > > Kafka Connect
> > > > Kafka Stream
> > > >
> > > > If vote for Core only loses (aka we will support subprojects) then:
> > > > We should look to add Kafka Rest
> > > >
> > > > And we should look to see how we can manage better govern and manage
> > > > submodules.
> > > >
> > > > A good example which id propose here is how some other communities in
> > > > Apache do this.
> > > >
> > > > Each Module has a Module Management Committee(MMC), this is like
> almost
> > > > the PMC but at a per module basis.
> > > >
> > > > This MMC should essentially hold the binding votes for that module.
> > > > The MMC should be made up of a single representative from each
> > > > organisation  (so no single organisation can fully veto the community
> > it
> > > > has to a genuine consenus)
> > > > The MMC requires at least 3 members (so there cant be a tied vote on
> 2)
> > > > For a new Module to be added a MMC committee should be sought
> > > > A new Module is only capable of being added if the above requirements
> > can
> > > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
> that
> > > > only actively support modules would be added
> > > >
> > > > The PMC reviews each module every 6months or Year. If MMC is
> inactive,
> > a
> > > > vote/call to find replacements if raised, if none are forthcoming
> > > dropping
> > > > the MMC to less than 3 then the module moves to "the attic" (very
> much
> > > like
> > > > apache attic but a little more aggressively)
> > > >
> > > > This way the PMC does not need to micro manage every module
> > > > We only add modules where some amount of active support and
> maintenance
> > > > and use is provided by the community
> > > > We have an automatic way to retire old or inactive projects.
> > > >
> > > > Thoughts?
> > > > Mike
> > > >
> > > >
> > > > 
> > > > From: Harsha Ch <harsha...@gmail.com>
> > > > Sent: Thursday, October 20, 2016 10:26 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > Jay,
> > > >   REST API is something every user is in need of. If the argument
> > is
> > > to
> > > > clone and write your  API, this will do a disservice to the users as
> > they
> > > > now have to choose one vs. others instead of keeping one API that is
> > > > supported in Kafka community.
> > > >
> > > > "Pre-emptively re-creating another
> > > > REST layer when it seems like we all quite agree on what needs to be
> > done
> > > > and we have an existing code base for HTTP/Kafka access that is
> heavily
> > > > used in production seems quite silly."
> > > >
> > > >Exactly our point. Why can't we develop this in Apache Kafka
> > > > community? Instead of us open sourcing another GitHub project and
> > > creating
> > > > a divide in users and another version of API. Let's build this in
> Kafka
> > > > Community and use the governance model that is proven to provide
> vendor
> > > > free user driven consensus features. The argument that is adding this
> > > REST
> > > > server to Kafka will affect the agility of the project doesn't mak
> > sense.
> > > >
> > > > It looks like your argument is either we develop all these small
> tools
> > or
> > > > none at all. We as a community need to look at supporting critical
> > > > tools/API. Instead of dividing this project into individual external
> > > > communities. We should build this as part of Kafka which best serves
> > the
> > > > needs of users.
&

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Chintalapani]

ther communities in
> > > Apache do this.
> > >
> > > Each Module has a Module Management Committee(MMC), this is like
almost
> > > the PMC but at a per module basis.
> > >
> > > This MMC should essentially hold the binding votes for that module.
> > > The MMC should be made up of a single representative from each
> > > organisation  (so no single organisation can fully veto the community
> it
> > > has to a genuine consenus)
> > > The MMC requires at least 3 members (so there cant be a tied vote on
2)
> > > For a new Module to be added a MMC committee should be sought
> > > A new Module is only capable of being added if the above requirements
> can
> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so
that
> > > only actively support modules would be added
> > >
> > > The PMC reviews each module every 6months or Year. If MMC is inactive,
> a
> > > vote/call to find replacements if raised, if none are forthcoming
> > dropping
> > > the MMC to less than 3 then the module moves to "the attic" (very much
> > like
> > > apache attic but a little more aggressively)
> > >
> > > This way the PMC does not need to micro manage every module
> > > We only add modules where some amount of active support and
maintenance
> > > and use is provided by the community
> > > We have an automatic way to retire old or inactive projects.
> > >
> > > Thoughts?
> > > Mike
> > >
> > >
> > > 
> > > From: Harsha Ch <harsha...@gmail.com>
> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > Jay,
> > >   REST API is something every user is in need of. If the argument
> is
> > to
> > > clone and write your  API, this will do a disservice to the users as
> they
> > > now have to choose one vs. others instead of keeping one API that is
> > > supported in Kafka community.
> > >
> > > "Pre-emptively re-creating another
> > > REST layer when it seems like we all quite agree on what needs to be
> done
> > > and we have an existing code base for HTTP/Kafka access that is
heavily
> > > used in production seems quite silly."
> > >
> > >Exactly our point. Why can't we develop this in Apache Kafka
> > > community? Instead of us open sourcing another GitHub project and
> > creating
> > > a divide in users and another version of API. Let's build this in
Kafka
> > > Community and use the governance model that is proven to provide
vendor
> > > free user driven consensus features. The argument that is adding this
> > REST
> > > server to Kafka will affect the agility of the project doesn't mak
> sense.
> > >
> > > It looks like your argument is either we develop all these small tools
> or
> > > none at all. We as a community need to look at supporting critical
> > > tools/API. Instead of dividing this project into individual external
> > > communities. We should build this as part of Kafka which best serves
> the
> > > needs of users.
> > > The Streams and Connect projects that were pushed into Kafka
> > could
> > > have been left in their own Github projects based on your arguments.
> What
> > > about the REST API is so different that such that it should stay out
of
> > the
> > > Kafka project? From my experience, more users are asking for the REST
> > API.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote:
> > >
> > > > I think the questions around governance make sense, I think we
should
> > > > really clarify that to make the process more clear so it can be
fully
> > > > inclusive.
> > > >
> > > > The idea that we should not collaborate on what is there now,
though,
> > > > because in the future we might disagree about direction does not
> really
> > > > make sense to me. If in the future we disagree, that is the beauty
of
> > > open
> > > > source, you can always fork off a copy of the code and start an
> > > independent
> > > > project either in Apache or elsewhere. Pre-emptively re-creating
> > another
> > > > 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Sriram Subramanian]

> > > the MMC to less than 3 then the module moves to "the attic" (very much
> > like
> > > apache attic but a little more aggressively)
> > >
> > > This way the PMC does not need to micro manage every module
> > > We only add modules where some amount of active support and maintenance
> > > and use is provided by the community
> > > We have an automatic way to retire old or inactive projects.
> > >
> > > Thoughts?
> > > Mike
> > >
> > >
> > > 
> > > From: Harsha Ch <harsha...@gmail.com>
> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > Jay,
> > >   REST API is something every user is in need of. If the argument
> is
> > to
> > > clone and write your  API, this will do a disservice to the users as
> they
> > > now have to choose one vs. others instead of keeping one API that is
> > > supported in Kafka community.
> > >
> > > "Pre-emptively re-creating another
> > > REST layer when it seems like we all quite agree on what needs to be
> done
> > > and we have an existing code base for HTTP/Kafka access that is heavily
> > > used in production seems quite silly."
> > >
> > >Exactly our point. Why can't we develop this in Apache Kafka
> > > community? Instead of us open sourcing another GitHub project and
> > creating
> > > a divide in users and another version of API. Let's build this in Kafka
> > > Community and use the governance model that is proven to provide vendor
> > > free user driven consensus features. The argument that is adding this
> > REST
> > > server to Kafka will affect the agility of the project doesn't mak
> sense.
> > >
> > > It looks like your argument is either we develop all these small tools
> or
> > > none at all. We as a community need to look at supporting critical
> > > tools/API. Instead of dividing this project into individual external
> > > communities. We should build this as part of Kafka which best serves
> the
> > > needs of users.
> > > The Streams and Connect projects that were pushed into Kafka
> > could
> > > have been left in their own Github projects based on your arguments.
> What
> > > about the REST API is so different that such that it should stay out of
> > the
> > > Kafka project? From my experience, more users are asking for the REST
> > API.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote:
> > >
> > > > I think the questions around governance make sense, I think we should
> > > > really clarify that to make the process more clear so it can be fully
> > > > inclusive.
> > > >
> > > > The idea that we should not collaborate on what is there now, though,
> > > > because in the future we might disagree about direction does not
> really
> > > > make sense to me. If in the future we disagree, that is the beauty of
> > > open
> > > > source, you can always fork off a copy of the code and start an
> > > independent
> > > > project either in Apache or elsewhere. Pre-emptively re-creating
> > another
> > > > REST layer when it seems like we all quite agree on what needs to be
> > done
> > > > and we have an existing code base for HTTP/kafka access that is
> heavily
> > > > used in production seems quite silly.
> > > >
> > > > Let me give some background on how I at least think about these
> things.
> > > > I've participated in open source projects out of LinkedIn via github
> as
> > > > well as via the ASF. I don't think there is a "right" answer to how
> to
> > do
> > > > these but rather some tradeoffs. We thought about this quite a lot in
> > the
> > > > context of Kafka based on the experience with the Hadoop ecosystem as
> > > well
> > > > as from other open source communities.
> > > >
> > > > There is a rich ecosystem around Kafka. Many of the projects are
> quite
> > > > small--single clients or tools that do a single thing well--and
> almost
> > > none
> > > > of them are top level ap

Re: [DISCUSS] KIP-80: Kafka REST Server

[Nacho Solis]

Are you saying Kafka REST is subjective but Kafka Streams and Kafka Connect
are not subjective?

> "there are likely places that can live without a rest proxy"

There are also places that can live without Kafka Streams and Kafka Connect.

Nacho

On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <j...@confluent.io> wrote:

> At the high level, I think ideally it makes sense to add a component to
> Apache Kafka if (1) it's widely needed and (2) it needs tight integration
> with Kafka core. For Kafka Stream, we do expect stream processing will be
> used widely in the future. Implementation wise, Kafka Stream only supports
> getting data from Kafka and leverages quite a few of the core
> functionalities in Kafka core. For example, it uses customized rebalance
> callback in the consumer and uses the compacted topic heavily. So, having
> Kafka Stream in the same repo makes it easier for testing when those core
> functionalities evolve over time. Kafka Connect is in the same situation.
>
> For rest proxy, whether it's widely used or not is going to be a bit
> subjective. However, there are likely places that can live without a rest
> proxy. The rest proxy is just a proxy for the regular clients and doesn't
> need to be tightly integrated with Kafka core. So, the case for including
> rest proxy in Apache Kafka is probably not as strong as Kafka Stream and
> Kafka Connect.
>
> Thanks,
>
> Jun
>
> On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <michael.pea...@ig.com>
> wrote:
>
> > So from my reading essentially the first question needs to answered/and
> > voted on is:
> >
> > Is Apache Kafka Community only about the Core or does the apache
> community
> > also support some subprojects (and just we need some better way to manage
> > this)
> >
> > If vote for Core only wins, then the following should be removed:
> > Kafka Connect
> > Kafka Stream
> >
> > If vote for Core only loses (aka we will support subprojects) then:
> > We should look to add Kafka Rest
> >
> > And we should look to see how we can manage better govern and manage
> > submodules.
> >
> > A good example which id propose here is how some other communities in
> > Apache do this.
> >
> > Each Module has a Module Management Committee(MMC), this is like almost
> > the PMC but at a per module basis.
> >
> > This MMC should essentially hold the binding votes for that module.
> > The MMC should be made up of a single representative from each
> > organisation  (so no single organisation can fully veto the community it
> > has to a genuine consenus)
> > The MMC requires at least 3 members (so there cant be a tied vote on 2)
> > For a new Module to be added a MMC committee should be sought
> > A new Module is only capable of being added if the above requirements can
> > be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> > only actively support modules would be added
> >
> > The PMC reviews each module every 6months or Year. If MMC is inactive, a
> > vote/call to find replacements if raised, if none are forthcoming
> dropping
> > the MMC to less than 3 then the module moves to "the attic" (very much
> like
> > apache attic but a little more aggressively)
> >
> > This way the PMC does not need to micro manage every module
> > We only add modules where some amount of active support and maintenance
> > and use is provided by the community
> > We have an automatic way to retire old or inactive projects.
> >
> > Thoughts?
> > Mike
> >
> >
> > 
> > From: Harsha Ch <harsha...@gmail.com>
> > Sent: Thursday, October 20, 2016 10:26 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > Jay,
> >   REST API is something every user is in need of. If the argument is
> to
> > clone and write your  API, this will do a disservice to the users as they
> > now have to choose one vs. others instead of keeping one API that is
> > supported in Kafka community.
> >
> > "Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existing code base for HTTP/Kafka access that is heavily
> > used in production seems quite silly."
> >
> >Exactly our point. Why can't we develop this in Apache Kafka
> > community? Instead of us open sourcing another GitHub project and
> creating
> > a divide in users and another version of API. Let's build this in Kafka
> > Commun

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jun Rao]

At the high level, I think ideally it makes sense to add a component to
Apache Kafka if (1) it's widely needed and (2) it needs tight integration
with Kafka core. For Kafka Stream, we do expect stream processing will be
used widely in the future. Implementation wise, Kafka Stream only supports
getting data from Kafka and leverages quite a few of the core
functionalities in Kafka core. For example, it uses customized rebalance
callback in the consumer and uses the compacted topic heavily. So, having
Kafka Stream in the same repo makes it easier for testing when those core
functionalities evolve over time. Kafka Connect is in the same situation.

For rest proxy, whether it's widely used or not is going to be a bit
subjective. However, there are likely places that can live without a rest
proxy. The rest proxy is just a proxy for the regular clients and doesn't
need to be tightly integrated with Kafka core. So, the case for including
rest proxy in Apache Kafka is probably not as strong as Kafka Stream and
Kafka Connect.

Thanks,

Jun

On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce <michael.pea...@ig.com>
wrote:

> So from my reading essentially the first question needs to answered/and
> voted on is:
>
> Is Apache Kafka Community only about the Core or does the apache community
> also support some subprojects (and just we need some better way to manage
> this)
>
> If vote for Core only wins, then the following should be removed:
> Kafka Connect
> Kafka Stream
>
> If vote for Core only loses (aka we will support subprojects) then:
> We should look to add Kafka Rest
>
> And we should look to see how we can manage better govern and manage
> submodules.
>
> A good example which id propose here is how some other communities in
> Apache do this.
>
> Each Module has a Module Management Committee(MMC), this is like almost
> the PMC but at a per module basis.
>
> This MMC should essentially hold the binding votes for that module.
> The MMC should be made up of a single representative from each
> organisation  (so no single organisation can fully veto the community it
> has to a genuine consenus)
> The MMC requires at least 3 members (so there cant be a tied vote on 2)
> For a new Module to be added a MMC committee should be sought
> A new Module is only capable of being added if the above requirements can
> be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> only actively support modules would be added
>
> The PMC reviews each module every 6months or Year. If MMC is inactive, a
> vote/call to find replacements if raised, if none are forthcoming dropping
> the MMC to less than 3 then the module moves to "the attic" (very much like
> apache attic but a little more aggressively)
>
> This way the PMC does not need to micro manage every module
> We only add modules where some amount of active support and maintenance
> and use is provided by the community
> We have an automatic way to retire old or inactive projects.
>
> Thoughts?
> Mike
>
>
> ________________________
> From: Harsha Ch <harsha...@gmail.com>
> Sent: Thursday, October 20, 2016 10:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Jay,
>   REST API is something every user is in need of. If the argument is to
> clone and write your  API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
>Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of the
> Kafka project? From my experi

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

ts.
> > Honestly,
> > > >though, if you go to the Kafka ecosystem page virtually none of
> the
> > > most
> > > >popular add-ons to Kafka are Apache projects. The most successful
> > > > things in
> > > >the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> > > other
> > > >clients, as well as the existing REST layer have succeeded at
> > > developing
> > > >communities that actively contribute and use these pieces and I
> > don't
> > > > know
> > > >that that is a bad thing unless that community proves to be
> > > uninclusive,
> > > >unresponsive, or goes in a bad technical direction--and those are
> > > > failure
> > > >modes that all open source efforts face.
> > > >3. We can do what I think makes the most sense and try to work
> with
> > > the
> > > >projects that exist in the ecosystem and if the project doesn't
> have
> > a
> > > >responsive community or wants to go in a different direction fork
> or
> > > >recreate that work.
> > > >
> > > > Of course any person can choose whatever of these options they want.
> > But
> > > > from my point of view, option (3) has been the path of the community
> so
> > > far
> > > > and I think it has been quite successful.
> > > >
> > > > -Jay
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> ka...@harsha.io
> > >
> > > > wrote:
> > > >
> > > > > Neha,
> > > > > "But I haven't seen any community emails or patches being submitted
> > by
> > > > you
> > > > > guys, so I'm wondering why you are concerned about whether the
> > > community
> > > > is
> > > > > open to accepting patches or not."
> > > > >
> > > > > I think you are talking about contributing patches to this
> repository
> > > > > right? https://github.com/confluentinc/kafka-rest . All I am
> saying
> > > > > the guidelines/governance model is not clear on the project and I
> > guess
> > > > its
> > > > > driven by opening a github issue request.  Its the repository owned
> > by
> > > > > confluent and as much I appreciate that the features we mentioned
> are
> > > in
> > > > > the roadmap and welcoming us to contribute to the project. It
> doesn't
> > > > > gurantee what we want to add in the furture will be in your
> roadmap.
> > > > >
> > > > > Hence the reason having it part of Kafka community will help a lot
> as
> > > > other
> > > > > users can participate in the discussions.  We are happy to drive
> any
> > > > > feature additions through KIPs which gives everyone a chance to
> > > > participate
> > > > > and add to the discussions.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > michael.pea...@ig.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > I agree on the governance comments whole heartedly.
> > > > > >
> > > > > > Also i agree about the contribution comments made earlier in the
> > > > thread,
> > > > > i
> > > > > > personally am less likely to spend any of mine, or give project
> > time
> > > > > within
> > > > > > my internal projects to developers contributing to another
> > commercial
> > > > > > companies project even so technically open source, as then there
> is
> > > > that
> > > > > > commercial companies interest will always prevail and essentially
> > can
> > > > > > always have a final vote where disagreement. Im sure they never
> > > intend
> > > > > to,
> > > > > > but there is that true reality. This is why we have community
> open
> > > > source
> > > > > > projects.
&g

Re: [DISCUSS] KIP-80: Kafka REST Server

[Michael Pearce]

Here is Ignite’s  - essentially idea is having different owners/maintainers per 
module/area that keep a check on that area.

https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute



On 10/21/16, 3:44 PM, "isma...@gmail.com on behalf of Ismael Juma" 
<isma...@gmail.com on behalf of ism...@juma.me.uk> wrote:

Hi Michael,

Can you please share which Apache projects have a MMC? I couldn't find
anything after a quick google.

Ismael

On Fri, Oct 21, 2016 at 7:28 AM, Michael Pearce <michael.pea...@ig.com>
wrote:

> So from my reading essentially the first question needs to answered/and
> voted on is:
>
> Is Apache Kafka Community only about the Core or does the apache community
> also support some subprojects (and just we need some better way to manage
> this)
>
> If vote for Core only wins, then the following should be removed:
> Kafka Connect
> Kafka Stream
>
> If vote for Core only loses (aka we will support subprojects) then:
> We should look to add Kafka Rest
>
> And we should look to see how we can manage better govern and manage
> submodules.
>
> A good example which id propose here is how some other communities in
> Apache do this.
>
> Each Module has a Module Management Committee(MMC), this is like almost
> the PMC but at a per module basis.
>
> This MMC should essentially hold the binding votes for that module.
> The MMC should be made up of a single representative from each
> organisation  (so no single organisation can fully veto the community it
> has to a genuine consenus)
> The MMC requires at least 3 members (so there cant be a tied vote on 2)
> For a new Module to be added a MMC committee should be sought
> A new Module is only capable of being added if the above requirements can
> be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> only actively support modules would be added
>
> The PMC reviews each module every 6months or Year. If MMC is inactive, a
> vote/call to find replacements if raised, if none are forthcoming dropping
> the MMC to less than 3 then the module moves to "the attic" (very much 
like
> apache attic but a little more aggressively)
>
> This way the PMC does not need to micro manage every module
> We only add modules where some amount of active support and maintenance
> and use is provided by the community
> We have an automatic way to retire old or inactive projects.
>
> Thoughts?
> Mike
>
>
> ________________________
    > From: Harsha Ch <harsha...@gmail.com>
> Sent: Thursday, October 20, 2016 10:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Jay,
>   REST API is something every user is in need of. If the argument is 
to
> clone and write your  API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
>Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of 
the
> Kafka project? From my experience, more users are asking for the REST API.
>
> Thanks,
> Harsha
>
>
>
>
>
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote:
>
&

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ismael Juma]

Hi Michael,

Can you please share which Apache projects have a MMC? I couldn't find
anything after a quick google.

Ismael

On Fri, Oct 21, 2016 at 7:28 AM, Michael Pearce <michael.pea...@ig.com>
wrote:

> So from my reading essentially the first question needs to answered/and
> voted on is:
>
> Is Apache Kafka Community only about the Core or does the apache community
> also support some subprojects (and just we need some better way to manage
> this)
>
> If vote for Core only wins, then the following should be removed:
> Kafka Connect
> Kafka Stream
>
> If vote for Core only loses (aka we will support subprojects) then:
> We should look to add Kafka Rest
>
> And we should look to see how we can manage better govern and manage
> submodules.
>
> A good example which id propose here is how some other communities in
> Apache do this.
>
> Each Module has a Module Management Committee(MMC), this is like almost
> the PMC but at a per module basis.
>
> This MMC should essentially hold the binding votes for that module.
> The MMC should be made up of a single representative from each
> organisation  (so no single organisation can fully veto the community it
> has to a genuine consenus)
> The MMC requires at least 3 members (so there cant be a tied vote on 2)
> For a new Module to be added a MMC committee should be sought
> A new Module is only capable of being added if the above requirements can
> be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> only actively support modules would be added
>
> The PMC reviews each module every 6months or Year. If MMC is inactive, a
> vote/call to find replacements if raised, if none are forthcoming dropping
> the MMC to less than 3 then the module moves to "the attic" (very much like
> apache attic but a little more aggressively)
>
> This way the PMC does not need to micro manage every module
> We only add modules where some amount of active support and maintenance
> and use is provided by the community
> We have an automatic way to retire old or inactive projects.
>
> Thoughts?
> Mike
>
>
> ________________________
> From: Harsha Ch <harsha...@gmail.com>
> Sent: Thursday, October 20, 2016 10:26 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> Jay,
>   REST API is something every user is in need of. If the argument is to
> clone and write your  API, this will do a disservice to the users as they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
>
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
>
>Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this REST
> server to Kafka will affect the agility of the project doesn't mak sense.
>
> It looks like your argument is either we develop all these small tools or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka could
> have been left in their own Github projects based on your arguments. What
> about the REST API is so different that such that it should stay out of the
> Kafka project? From my experience, more users are asking for the REST API.
>
> Thanks,
> Harsha
>
>
>
>
>
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote:
>
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not really
> > make sense to me. If in the future we disagree, that is the beauty of
> open
> > source, you can always fork off a copy of the code and start an
> independent
> > project either in Apache or elsewhere. Pre-emptively re-creating another
> > REST layer when it seems like we all quite agree on what needs to be done
> > and we have an existi

Re: [DISCUSS] KIP-80: Kafka REST Server

[Edoardo Comar]

Harsha Ch <harsha...@gmail.com> wrote on 20/10/2016 22:26:53:

> The Streams and Connect projects that were pushed into Kafka 
could
> have been left in their own Github projects based on your arguments. 
What
> about the REST API is so different that such that it should stay out of 
the
> Kafka project? From my experience, more users are asking for the REST 
API.

Thanks Harsha
for keeping pushing this issue.
Your experience about users matches ours !

cheers
Edo

--
Edoardo Comar
IBM MessageHub
eco...@uk.ibm.com
IBM UK Ltd, Hursley Park, SO21 2JN

IBM United Kingdom Limited Registered in England and Wales with number 
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 
3AU

Harsha Ch <harsha...@gmail.com> wrote on 20/10/2016 22:26:53:

> From: Harsha Ch <harsha...@gmail.com>
> To: dev@kafka.apache.org
> Date: 20/10/2016 22:32
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> 
> Jay,
>   REST API is something every user is in need of. If the argument is 
to
> clone and write your  API, this will do a disservice to the users as 
they
> now have to choose one vs. others instead of keeping one API that is
> supported in Kafka community.
> 
> "Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be 
done
> and we have an existing code base for HTTP/Kafka access that is heavily
> used in production seems quite silly."
> 
>Exactly our point. Why can't we develop this in Apache Kafka
> community? Instead of us open sourcing another GitHub project and 
creating
> a divide in users and another version of API. Let's build this in Kafka
> Community and use the governance model that is proven to provide vendor
> free user driven consensus features. The argument that is adding this 
REST
> server to Kafka will affect the agility of the project doesn't mak 
sense.
> 
> It looks like your argument is either we develop all these small tools 
or
> none at all. We as a community need to look at supporting critical
> tools/API. Instead of dividing this project into individual external
> communities. We should build this as part of Kafka which best serves the
> needs of users.
> The Streams and Connect projects that were pushed into Kafka 
could
> have been left in their own Github projects based on your arguments. 
What
> about the REST API is so different that such that it should stay out of 
the
> Kafka project? From my experience, more users are asking for the REST 
API.
> 
> Thanks,
> Harsha
> 
> 
> 
> 
> 
> On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote:
> 
> > I think the questions around governance make sense, I think we should
> > really clarify that to make the process more clear so it can be fully
> > inclusive.
> >
> > The idea that we should not collaborate on what is there now, though,
> > because in the future we might disagree about direction does not 
really
> > make sense to me. If in the future we disagree, that is the beauty of 
open
> > source, you can always fork off a copy of the code and start an 
independent
> > project either in Apache or elsewhere. Pre-emptively re-creating 
another
> > REST layer when it seems like we all quite agree on what needs to be 
done
> > and we have an existing code base for HTTP/kafka access that is 
heavily
> > used in production seems quite silly.
> >
> > Let me give some background on how I at least think about these 
things.
> > I've participated in open source projects out of LinkedIn via github 
as
> > well as via the ASF. I don't think there is a "right" answer to how to 
do
> > these but rather some tradeoffs. We thought about this quite a lot in 
the
> > context of Kafka based on the experience with the Hadoop ecosystem as 
well
> > as from other open source communities.
> >
> > There is a rich ecosystem around Kafka. Many of the projects are quite
> > small--single clients or tools that do a single thing well--and almost 
none
> > of them are top level apache projects. I don't think trying to force 
each
> > of these to turn into independent Apache projects is necessarily the 
best
> > thing for the ecosystem.
> >
> > My observation of how this can go wrong is really what I think has 
happened
> > to the Hadoop ecosystem. There you see quite a zoo of projects which 
all
> > drift apart and don't quite work together well. Coordinating even 
simple
> > changes and standardization across these is exceptionally difficult. 
The
> > result is a bit of a mess for users--the pieces just don't really come
> > together very well. Th

Re: [DISCUSS] KIP-80: Kafka REST Server

[Michael Pearce]

So from my reading essentially the first question needs to answered/and voted 
on is:

Is Apache Kafka Community only about the Core or does the apache community also 
support some subprojects (and just we need some better way to manage this)

If vote for Core only wins, then the following should be removed:
Kafka Connect
Kafka Stream

If vote for Core only loses (aka we will support subprojects) then:
We should look to add Kafka Rest

And we should look to see how we can manage better govern and manage submodules.

A good example which id propose here is how some other communities in Apache do 
this.

Each Module has a Module Management Committee(MMC), this is like almost the PMC 
but at a per module basis.

This MMC should essentially hold the binding votes for that module.
The MMC should be made up of a single representative from each organisation  
(so no single organisation can fully veto the community it has to a genuine 
consenus)
The MMC requires at least 3 members (so there cant be a tied vote on 2)
For a new Module to be added a MMC committee should be sought
A new Module is only capable of being added if the above requirements can be 
met (e.g. 3 people wishing to step up, from 3 organisations) so that only 
actively support modules would be added

The PMC reviews each module every 6months or Year. If MMC is inactive, a 
vote/call to find replacements if raised, if none are forthcoming dropping the 
MMC to less than 3 then the module moves to "the attic" (very much like apache 
attic but a little more aggressively)

This way the PMC does not need to micro manage every module
We only add modules where some amount of active support and maintenance and use 
is provided by the community
We have an automatic way to retire old or inactive projects.

Thoughts?
Mike



From: Harsha Ch <harsha...@gmail.com>
Sent: Thursday, October 20, 2016 10:26 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server

Jay,
  REST API is something every user is in need of. If the argument is to
clone and write your  API, this will do a disservice to the users as they
now have to choose one vs. others instead of keeping one API that is
supported in Kafka community.

"Pre-emptively re-creating another
REST layer when it seems like we all quite agree on what needs to be done
and we have an existing code base for HTTP/Kafka access that is heavily
used in production seems quite silly."

   Exactly our point. Why can't we develop this in Apache Kafka
community? Instead of us open sourcing another GitHub project and creating
a divide in users and another version of API. Let's build this in Kafka
Community and use the governance model that is proven to provide vendor
free user driven consensus features. The argument that is adding this REST
server to Kafka will affect the agility of the project doesn't mak sense.

It looks like your argument is either we develop all these small tools or
none at all. We as a community need to look at supporting critical
tools/API. Instead of dividing this project into individual external
communities. We should build this as part of Kafka which best serves the
needs of users.
The Streams and Connect projects that were pushed into Kafka could
have been left in their own Github projects based on your arguments. What
about the REST API is so different that such that it should stay out of the
Kafka project? From my experience, more users are asking for the REST API.

Thanks,
Harsha





On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io> wrote:

> I think the questions around governance make sense, I think we should
> really clarify that to make the process more clear so it can be fully
> inclusive.
>
> The idea that we should not collaborate on what is there now, though,
> because in the future we might disagree about direction does not really
> make sense to me. If in the future we disagree, that is the beauty of open
> source, you can always fork off a copy of the code and start an independent
> project either in Apache or elsewhere. Pre-emptively re-creating another
> REST layer when it seems like we all quite agree on what needs to be done
> and we have an existing code base for HTTP/kafka access that is heavily
> used in production seems quite silly.
>
> Let me give some background on how I at least think about these things.
> I've participated in open source projects out of LinkedIn via github as
> well as via the ASF. I don't think there is a "right" answer to how to do
> these but rather some tradeoffs. We thought about this quite a lot in the
> context of Kafka based on the experience with the Hadoop ecosystem as well
> as from other open source communities.
>
> There is a rich ecosystem around Kafka. Many of the projects are quite
> small--single clients or tools that do a single thing well--an

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jason Gustafson]

admap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions.  We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > 
> > > > From: Andrew Schofield <andrew_schofield_j...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable  in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works  nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
> > > > the way that there is with Kafka. So, you could say that it's clear
> > that
> > > > fewer people are interested, but I think  the whole governance thing
> > is a
> > > > big barrier to engagement. And it's looking like it's getting out of
> > > date.
> > > >
> > > > In technical terms, I can think of two big improvements to the REST
> > > proxy.
> > > > First, it needs to use the new consumer API so that it's possible to
> > > secure
> > > > connections between the REST proxy and Kafka. I don't care too much
> > which
> > > > method calls it uses actually  uses to consume messages, but I do
> care
> > > that
> > > > I cannot secure connections because of network security rules.
> Second,
> > > > there's an affinity between a consumer and the instance of the REST
> > proxy
> > > > to which it first connected. Kafka itself avoids this kind of
> affinity
> > > for
> > > > good reason, and in the name of availability the REST proxy should
> too.
> > > 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Ch]

single purpose tools, especially if you hope
> for these to come together into a cohesive toolset across multiple such
> tools. Much of what Apache does--create a collective decision making
> process for resolving disagreement, help to trademark and protect the marks
> of the project, etc just isn't that relevant for simple single-purpose
> tools.
>
> So my take is there are a couple of options:
>
>1. We can try to put all the small tools into the Apache Project. I
>think this is not the right approach as there is simply too many of
> them,
>many in different languages, serving different protocols, integrating
> with
>particular systems, and a single community can't effectively maintain
> them
>all. Doing this would significantly slow the progress of the Kafka
> project.
>As a protocol for messaging, I don't really see a case for including
> REST
>but not MQTT or AMQP which are technically much better suited to
> messaging
>and both are widely used for that.
>2. We can treat ecosystem projects that aren't top level Apache projects
>as invalid and try to recreate them all as Apache projects. Honestly,
>though, if you go to the Kafka ecosystem page virtually none of the most
>popular add-ons to Kafka are Apache projects. The most successful
> things in
>the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of other
>clients, as well as the existing REST layer have succeeded at developing
>communities that actively contribute and use these pieces and I don't
> know
>that that is a bad thing unless that community proves to be uninclusive,
>unresponsive, or goes in a bad technical direction--and those are
> failure
>modes that all open source efforts face.
>3. We can do what I think makes the most sense and try to work with the
>projects that exist in the ecosystem and if the project doesn't have a
>responsive community or wants to go in a different direction fork or
>recreate that work.
>
> Of course any person can choose whatever of these options they want. But
> from my point of view, option (3) has been the path of the community so far
> and I think it has been quite successful.
>
> -Jay
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
>
> > Neha,
> > "But I haven't seen any community emails or patches being submitted by
> you
> > guys, so I'm wondering why you are concerned about whether the community
> is
> > open to accepting patches or not."
> >
> > I think you are talking about contributing patches to this repository
> > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > the guidelines/governance model is not clear on the project and I guess
> its
> > driven by opening a github issue request.  Its the repository owned by
> > confluent and as much I appreciate that the features we mentioned are in
> > the roadmap and welcoming us to contribute to the project. It doesn't
> > gurantee what we want to add in the furture will be in your roadmap.
> >
> > Hence the reason having it part of Kafka community will help a lot as
> other
> > users can participate in the discussions.  We are happy to drive any
> > feature additions through KIPs which gives everyone a chance to
> participate
> > and add to the discussions.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com>
> > wrote:
> >
> > > +1
> > >
> > > I agree on the governance comments whole heartedly.
> > >
> > > Also i agree about the contribution comments made earlier in the
> thread,
> > i
> > > personally am less likely to spend any of mine, or give project time
> > within
> > > my internal projects to developers contributing to another commercial
> > > companies project even so technically open source, as then there is
> that
> > > commercial companies interest will always prevail and essentially can
> > > always have a final vote where disagreement. Im sure they never intend
> > to,
> > > but there is that true reality. This is why we have community open
> source
> > > projects.
> > >
> > > I can find many different implementations now of a rest endpoint on
> > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> > > their implementation. By making / providing one this would bring
> together
> > > these solutions, unifying those developers and also bringing the best
> of
&g

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Chintalapani]

es or not."
> > > >
> > > > I think you are talking about contributing patches to this repository
> > > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > > the guidelines/governance model is not clear on the project and I
> guess
> > > its
> > > > driven by opening a github issue request.  Its the repository owned
> by
> > > > confluent and as much I appreciate that the features we mentioned are
> > in
> > > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > > gurantee what we want to add in the furture will be in your roadmap.
> > > >
> > > > Hence the reason having it part of Kafka community will help a lot as
> > > other
> > > > users can participate in the discussions.  We are happy to drive any
> > > > feature additions through KIPs which gives everyone a chance to
> > > participate
> > > > and add to the discussions.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> michael.pea...@ig.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > I agree on the governance comments whole heartedly.
> > > > >
> > > > > Also i agree about the contribution comments made earlier in the
> > > thread,
> > > > i
> > > > > personally am less likely to spend any of mine, or give project
> time
> > > > within
> > > > > my internal projects to developers contributing to another
> commercial
> > > > > companies project even so technically open source, as then there is
> > > that
> > > > > commercial companies interest will always prevail and essentially
> can
> > > > > always have a final vote where disagreement. Im sure they never
> > intend
> > > > to,
> > > > > but there is that true reality. This is why we have community open
> > > source
> > > > > projects.
> > > > >
> > > > > I can find many different implementations now of a rest endpoint on
> > > > > GitHub, BitBucket etc. Each one has their benefits and
> disadvantages
> > in
> > > > > their implementation. By making / providing one this would bring
> > > together
> > > > > these solutions, unifying those developers and also bringing the
> best
> > > of
> > > > > all.
> > > > >
> > > > > I understand the concern on the community burden adding/supporting
> > more
> > > > > surface area for every client. But something like REST is universal
> > and
> > > > > worthy to be owned by the community.
> > > > >
> > > > > Mike
> > > > >
> > > > >
> > > > > 
> > > > > From: Andrew Schofield <andrew_schofield_j...@outlook.com>
> > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >
> > > > > There's a massive difference between the governance of Kafka and
> the
> > > > > governance of the REST proxy.
> > > > >
> > > > > In Kafka, there is a broad community of people contributing their
> > > > opinions
> > > > > about future enhancements in the form of KIPs. There's some really
> > deep
> > > > > consideration that goes into some of the trickier KIPs. There are
> > > people
> > > > > outside Confluent deeply knowledgeable  in Kafka and building the
> > > > > reputations to become committers. I get the impression that the
> > roadmap
> > > > of
> > > > > Kafka is not really community-owned (what's the big feature for
> Kafka
> > > > 0.11,
> > > > > for example), but the conveyor belt of smaller features in the form
> > of
> > > > KIPs
> > > > > works  nicely. It's a good example of open-source working well.
> > > > >
> > > > > The equivalent for the REST proxy is basically issues on GitHub.
> The
> > > > > roadmap is less clear. There's not really a community properly
> > engaged
> > > in
> > > > > the way that there is wit

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jungtaek Lim]

gt; messaging
> >and both are widely used for that.
> >2. We can treat ecosystem projects that aren't top level Apache
> projects
> >as invalid and try to recreate them all as Apache projects. Honestly,
> >though, if you go to the Kafka ecosystem page virtually none of the
> most
> >popular add-ons to Kafka are Apache projects. The most successful
> > things in
> >the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of
> other
> >clients, as well as the existing REST layer have succeeded at
> developing
> >communities that actively contribute and use these pieces and I don't
> > know
> >that that is a bad thing unless that community proves to be
> uninclusive,
> >unresponsive, or goes in a bad technical direction--and those are
> > failure
> >modes that all open source efforts face.
> >3. We can do what I think makes the most sense and try to work with
> the
> >projects that exist in the ecosystem and if the project doesn't have
a
> >responsive community or wants to go in a different direction fork or
> >recreate that work.
> >
> > Of course any person can choose whatever of these options they want. But
> > from my point of view, option (3) has been the path of the community so
> far
> > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I
guess
> > its
> > > driven by opening a github issue request.  Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions.  We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another
commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially
can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the
best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > 
> > > > From: Andrew Schofield <andrew_sch

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

t; > and I think it has been quite successful.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> >
> > > Neha,
> > > "But I haven't seen any community emails or patches being submitted by
> > you
> > > guys, so I'm wondering why you are concerned about whether the
> community
> > is
> > > open to accepting patches or not."
> > >
> > > I think you are talking about contributing patches to this repository
> > > right? https://github.com/confluentinc/kafka-rest . All I am saying
> > > the guidelines/governance model is not clear on the project and I guess
> > its
> > > driven by opening a github issue request.  Its the repository owned by
> > > confluent and as much I appreciate that the features we mentioned are
> in
> > > the roadmap and welcoming us to contribute to the project. It doesn't
> > > gurantee what we want to add in the furture will be in your roadmap.
> > >
> > > Hence the reason having it part of Kafka community will help a lot as
> > other
> > > users can participate in the discussions.  We are happy to drive any
> > > feature additions through KIPs which gives everyone a chance to
> > participate
> > > and add to the discussions.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > I agree on the governance comments whole heartedly.
> > > >
> > > > Also i agree about the contribution comments made earlier in the
> > thread,
> > > i
> > > > personally am less likely to spend any of mine, or give project time
> > > within
> > > > my internal projects to developers contributing to another commercial
> > > > companies project even so technically open source, as then there is
> > that
> > > > commercial companies interest will always prevail and essentially can
> > > > always have a final vote where disagreement. Im sure they never
> intend
> > > to,
> > > > but there is that true reality. This is why we have community open
> > source
> > > > projects.
> > > >
> > > > I can find many different implementations now of a rest endpoint on
> > > > GitHub, BitBucket etc. Each one has their benefits and disadvantages
> in
> > > > their implementation. By making / providing one this would bring
> > together
> > > > these solutions, unifying those developers and also bringing the best
> > of
> > > > all.
> > > >
> > > > I understand the concern on the community burden adding/supporting
> more
> > > > surface area for every client. But something like REST is universal
> and
> > > > worthy to be owned by the community.
> > > >
> > > > Mike
> > > >
> > > >
> > > > 
> > > > From: Andrew Schofield <andrew_schofield_j...@outlook.com>
> > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > >
> > > > There's a massive difference between the governance of Kafka and the
> > > > governance of the REST proxy.
> > > >
> > > > In Kafka, there is a broad community of people contributing their
> > > opinions
> > > > about future enhancements in the form of KIPs. There's some really
> deep
> > > > consideration that goes into some of the trickier KIPs. There are
> > people
> > > > outside Confluent deeply knowledgeable  in Kafka and building the
> > > > reputations to become committers. I get the impression that the
> roadmap
> > > of
> > > > Kafka is not really community-owned (what's the big feature for Kafka
> > > 0.11,
> > > > for example), but the conveyor belt of smaller features in the form
> of
> > > KIPs
> > > > works  nicely. It's a good example of open-source working well.
> > > >
> > > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > > roadmap is less clear. There's not really a community properly
> engaged
> > in
&

Re: [DISCUSS] KIP-80: Kafka REST Server

[Nacho Solis]

.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com>
> > wrote:
> >
> > > +1
> > >
> > > I agree on the governance comments whole heartedly.
> > >
> > > Also i agree about the contribution comments made earlier in the
> thread,
> > i
> > > personally am less likely to spend any of mine, or give project time
> > within
> > > my internal projects to developers contributing to another commercial
> > > companies project even so technically open source, as then there is
> that
> > > commercial companies interest will always prevail and essentially can
> > > always have a final vote where disagreement. Im sure they never intend
> > to,
> > > but there is that true reality. This is why we have community open
> source
> > > projects.
> > >
> > > I can find many different implementations now of a rest endpoint on
> > > GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> > > their implementation. By making / providing one this would bring
> together
> > > these solutions, unifying those developers and also bringing the best
> of
> > > all.
> > >
> > > I understand the concern on the community burden adding/supporting more
> > > surface area for every client. But something like REST is universal and
> > > worthy to be owned by the community.
> > >
> > > Mike
> > >
> > >
> > > 
> > > From: Andrew Schofield <andrew_schofield_j...@outlook.com>
> > > Sent: Saturday, October 8, 2016 1:19 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > There's a massive difference between the governance of Kafka and the
> > > governance of the REST proxy.
> > >
> > > In Kafka, there is a broad community of people contributing their
> > opinions
> > > about future enhancements in the form of KIPs. There's some really deep
> > > consideration that goes into some of the trickier KIPs. There are
> people
> > > outside Confluent deeply knowledgeable  in Kafka and building the
> > > reputations to become committers. I get the impression that the roadmap
> > of
> > > Kafka is not really community-owned (what's the big feature for Kafka
> > 0.11,
> > > for example), but the conveyor belt of smaller features in the form of
> > KIPs
> > > works  nicely. It's a good example of open-source working well.
> > >
> > > The equivalent for the REST proxy is basically issues on GitHub. The
> > > roadmap is less clear. There's not really a community properly engaged
> in
> > > the way that there is with Kafka. So, you could say that it's clear
> that
> > > fewer people are interested, but I think  the whole governance thing
> is a
> > > big barrier to engagement. And it's looking like it's getting out of
> > date.
> > >
> > > In technical terms, I can think of two big improvements to the REST
> > proxy.
> > > First, it needs to use the new consumer API so that it's possible to
> > secure
> > > connections between the REST proxy and Kafka. I don't care too much
> which
> > > method calls it uses actually  uses to consume messages, but I do care
> > that
> > > I cannot secure connections because of network security rules. Second,
> > > there's an affinity between a consumer and the instance of the REST
> proxy
> > > to which it first connected. Kafka itself avoids this kind of affinity
> > for
> > > good reason, and in the name of availability the REST proxy should too.
> > > These are natural KIPs.
> > >
> > > I think it would be good to have the code for the REST proxy
> contributed
> > > to Apache Kafka so that it would be able to be developed in the same
> way.
> > >
> > > Andrew Schofield
> > >
> > > From: Suresh Srinivas <sur...@hortonworks.com>
> > > Sent: 07 October 2016 22:41:52
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > >
> > > ASF already gives us a clear framework and governance model for
> community
> > > development. This is already understood by the people contributing to
> > > Apache Kafka project, and they are the same people who want to
> contribute
> > > to the REST server capability as well. Everyone is 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

ants to go in a different direction fork or
   recreate that work.

Of course any person can choose whatever of these options they want. But
from my point of view, option (3) has been the path of the community so far
and I think it has been quite successful.

-Jay











On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> Neha,
> "But I haven't seen any community emails or patches being submitted by you
> guys, so I'm wondering why you are concerned about whether the community is
> open to accepting patches or not."
>
> I think you are talking about contributing patches to this repository
> right? https://github.com/confluentinc/kafka-rest . All I am saying
> the guidelines/governance model is not clear on the project and I guess its
> driven by opening a github issue request.  Its the repository owned by
> confluent and as much I appreciate that the features we mentioned are in
> the roadmap and welcoming us to contribute to the project. It doesn't
> gurantee what we want to add in the furture will be in your roadmap.
>
> Hence the reason having it part of Kafka community will help a lot as other
> users can participate in the discussions.  We are happy to drive any
> feature additions through KIPs which gives everyone a chance to participate
> and add to the discussions.
>
> Thanks,
> Harsha
>
> On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com>
> wrote:
>
> > +1
> >
> > I agree on the governance comments whole heartedly.
> >
> > Also i agree about the contribution comments made earlier in the thread,
> i
> > personally am less likely to spend any of mine, or give project time
> within
> > my internal projects to developers contributing to another commercial
> > companies project even so technically open source, as then there is that
> > commercial companies interest will always prevail and essentially can
> > always have a final vote where disagreement. Im sure they never intend
> to,
> > but there is that true reality. This is why we have community open source
> > projects.
> >
> > I can find many different implementations now of a rest endpoint on
> > GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> > their implementation. By making / providing one this would bring together
> > these solutions, unifying those developers and also bringing the best of
> > all.
> >
> > I understand the concern on the community burden adding/supporting more
> > surface area for every client. But something like REST is universal and
> > worthy to be owned by the community.
> >
> > Mike
> >
> >
> > 
> > From: Andrew Schofield <andrew_schofield_j...@outlook.com>
> > Sent: Saturday, October 8, 2016 1:19 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > There's a massive difference between the governance of Kafka and the
> > governance of the REST proxy.
> >
> > In Kafka, there is a broad community of people contributing their
> opinions
> > about future enhancements in the form of KIPs. There's some really deep
> > consideration that goes into some of the trickier KIPs. There are people
> > outside Confluent deeply knowledgeable  in Kafka and building the
> > reputations to become committers. I get the impression that the roadmap
> of
> > Kafka is not really community-owned (what's the big feature for Kafka
> 0.11,
> > for example), but the conveyor belt of smaller features in the form of
> KIPs
> > works  nicely. It's a good example of open-source working well.
> >
> > The equivalent for the REST proxy is basically issues on GitHub. The
> > roadmap is less clear. There's not really a community properly engaged in
> > the way that there is with Kafka. So, you could say that it's clear that
> > fewer people are interested, but I think  the whole governance thing is a
> > big barrier to engagement. And it's looking like it's getting out of
> date.
> >
> > In technical terms, I can think of two big improvements to the REST
> proxy.
> > First, it needs to use the new consumer API so that it's possible to
> secure
> > connections between the REST proxy and Kafka. I don't care too much which
> > method calls it uses actually  uses to consume messages, but I do care
> that
> > I cannot secure connections because of network security rules. Second,
> > there's an affinity between a consumer and the instance of the REST proxy
> > to which it first connected. Kafka itself avoids this kind of affinity
> for
> >

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Chintalapani]

Neha,
"But I haven't seen any community emails or patches being submitted by you
guys, so I'm wondering why you are concerned about whether the community is
open to accepting patches or not."

I think you are talking about contributing patches to this repository
right? https://github.com/confluentinc/kafka-rest . All I am saying
the guidelines/governance model is not clear on the project and I guess its
driven by opening a github issue request.  Its the repository owned by
confluent and as much I appreciate that the features we mentioned are in
the roadmap and welcoming us to contribute to the project. It doesn't
gurantee what we want to add in the furture will be in your roadmap.

Hence the reason having it part of Kafka community will help a lot as other
users can participate in the discussions.  We are happy to drive any
feature additions through KIPs which gives everyone a chance to participate
and add to the discussions.

Thanks,
Harsha

On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <michael.pea...@ig.com>
wrote:

> +1
>
> I agree on the governance comments whole heartedly.
>
> Also i agree about the contribution comments made earlier in the thread, i
> personally am less likely to spend any of mine, or give project time within
> my internal projects to developers contributing to another commercial
> companies project even so technically open source, as then there is that
> commercial companies interest will always prevail and essentially can
> always have a final vote where disagreement. Im sure they never intend to,
> but there is that true reality. This is why we have community open source
> projects.
>
> I can find many different implementations now of a rest endpoint on
> GitHub, BitBucket etc. Each one has their benefits and disadvantages in
> their implementation. By making / providing one this would bring together
> these solutions, unifying those developers and also bringing the best of
> all.
>
> I understand the concern on the community burden adding/supporting more
> surface area for every client. But something like REST is universal and
> worthy to be owned by the community.
>
> Mike
>
>
> 
> From: Andrew Schofield <andrew_schofield_j...@outlook.com>
> Sent: Saturday, October 8, 2016 1:19 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> There's a massive difference between the governance of Kafka and the
> governance of the REST proxy.
>
> In Kafka, there is a broad community of people contributing their opinions
> about future enhancements in the form of KIPs. There's some really deep
> consideration that goes into some of the trickier KIPs. There are people
> outside Confluent deeply knowledgeable  in Kafka and building the
> reputations to become committers. I get the impression that the roadmap of
> Kafka is not really community-owned (what's the big feature for Kafka 0.11,
> for example), but the conveyor belt of smaller features in the form of KIPs
> works  nicely. It's a good example of open-source working well.
>
> The equivalent for the REST proxy is basically issues on GitHub. The
> roadmap is less clear. There's not really a community properly engaged in
> the way that there is with Kafka. So, you could say that it's clear that
> fewer people are interested, but I think  the whole governance thing is a
> big barrier to engagement. And it's looking like it's getting out of date.
>
> In technical terms, I can think of two big improvements to the REST proxy.
> First, it needs to use the new consumer API so that it's possible to secure
> connections between the REST proxy and Kafka. I don't care too much which
> method calls it uses actually  uses to consume messages, but I do care that
> I cannot secure connections because of network security rules. Second,
> there's an affinity between a consumer and the instance of the REST proxy
> to which it first connected. Kafka itself avoids this kind of affinity for
> good reason, and in the name of availability the REST proxy should too.
> These are natural KIPs.
>
> I think it would be good to have the code for the REST proxy contributed
> to Apache Kafka so that it would be able to be developed in the same way.
>
> Andrew Schofield
>
> From: Suresh Srinivas <sur...@hortonworks.com>
> Sent: 07 October 2016 22:41:52
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
>
> ASF already gives us a clear framework and governance model for community
> development. This is already understood by the people contributing to
> Apache Kafka project, and they are the same people who want to contribute
> to the REST server capability as well. Everyone is in agreement on the
> need for collaborating o

Re: [DISCUSS] KIP-80: Kafka REST Server

[Michael Pearce]

+1

I agree on the governance comments whole heartedly.

Also i agree about the contribution comments made earlier in the thread, i 
personally am less likely to spend any of mine, or give project time within my 
internal projects to developers contributing to another commercial companies 
project even so technically open source, as then there is that commercial 
companies interest will always prevail and essentially can always have a final 
vote where disagreement. Im sure they never intend to, but there is that true 
reality. This is why we have community open source projects.

I can find many different implementations now of a rest endpoint on GitHub, 
BitBucket etc. Each one has their benefits and disadvantages in their 
implementation. By making / providing one this would bring together these 
solutions, unifying those developers and also bringing the best of all.

I understand the concern on the community burden adding/supporting more surface 
area for every client. But something like REST is universal and worthy to be 
owned by the community.

Mike



From: Andrew Schofield <andrew_schofield_j...@outlook.com>
Sent: Saturday, October 8, 2016 1:19 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server

There's a massive difference between the governance of Kafka and the governance 
of the REST proxy.

In Kafka, there is a broad community of people contributing their opinions 
about future enhancements in the form of KIPs. There's some really deep 
consideration that goes into some of the trickier KIPs. There are people 
outside Confluent deeply knowledgeable  in Kafka and building the reputations 
to become committers. I get the impression that the roadmap of Kafka is not 
really community-owned (what's the big feature for Kafka 0.11, for example), 
but the conveyor belt of smaller features in the form of KIPs works  nicely. 
It's a good example of open-source working well.

The equivalent for the REST proxy is basically issues on GitHub. The roadmap is 
less clear. There's not really a community properly engaged in the way that 
there is with Kafka. So, you could say that it's clear that fewer people are 
interested, but I think  the whole governance thing is a big barrier to 
engagement. And it's looking like it's getting out of date.

In technical terms, I can think of two big improvements to the REST proxy. 
First, it needs to use the new consumer API so that it's possible to secure 
connections between the REST proxy and Kafka. I don't care too much which 
method calls it uses actually  uses to consume messages, but I do care that I 
cannot secure connections because of network security rules. Second, there's an 
affinity between a consumer and the instance of the REST proxy to which it 
first connected. Kafka itself avoids this kind of affinity for good reason, and 
in the name of availability the REST proxy should too. These are natural KIPs.

I think it would be good to have the code for the REST proxy contributed to 
Apache Kafka so that it would be able to be developed in the same way.

Andrew Schofield

From: Suresh Srinivas <sur...@hortonworks.com>
Sent: 07 October 2016 22:41:52
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server

ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.

As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.

Regards,
Suresh

On 10/6/16, 8:32 AM, "Ofir Manor" <ofir.ma...@equalum.io> wrote:

>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka  community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that

Re: [DISCUSS] KIP-80: Kafka REST Server

[Andrew Schofield]

There's a massive difference between the governance of Kafka and the governance 
of the REST proxy.

In Kafka, there is a broad community of people contributing their opinions 
about future enhancements in the form of KIPs. There's some really deep 
consideration that goes into some of the trickier KIPs. There are people 
outside Confluent deeply knowledgeable  in Kafka and building the reputations 
to become committers. I get the impression that the roadmap of Kafka is not 
really community-owned (what's the big feature for Kafka 0.11, for example), 
but the conveyor belt of smaller features in the form of KIPs works  nicely. 
It's a good example of open-source working well.

The equivalent for the REST proxy is basically issues on GitHub. The roadmap is 
less clear. There's not really a community properly engaged in the way that 
there is with Kafka. So, you could say that it's clear that fewer people are 
interested, but I think  the whole governance thing is a big barrier to 
engagement. And it's looking like it's getting out of date.

In technical terms, I can think of two big improvements to the REST proxy. 
First, it needs to use the new consumer API so that it's possible to secure 
connections between the REST proxy and Kafka. I don't care too much which 
method calls it uses actually  uses to consume messages, but I do care that I 
cannot secure connections because of network security rules. Second, there's an 
affinity between a consumer and the instance of the REST proxy to which it 
first connected. Kafka itself avoids this kind of affinity for good reason, and 
in the name of availability the REST proxy should too. These are natural KIPs.

I think it would be good to have the code for the REST proxy contributed to 
Apache Kafka so that it would be able to be developed in the same way.

Andrew Schofield
  
From: Suresh Srinivas <sur...@hortonworks.com>
Sent: 07 October 2016 22:41:52
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
    
ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.

As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.

Regards,
Suresh

On 10/6/16, 8:32 AM, "Ofir Manor" <ofir.ma...@equalum.io> wrote:

>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka  community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that, existing REST API project doesn't support new client API
>and
>>> hence there is no security support either.
>
>This might be easy to fix. Maybe Confluent / kafka-rest community can
>clarify that - what is their contribution policy, dev style, roadmap etc.
>If they want, they can make an effort to encourage participation from
>people outside Confluent (easily accept contributions, invite external
>commiters or have open dev process similar to Apache Kafka etc), as there
>is definitely seems to be some interest on the list. That might clear the
>community concern and help kafka-rest project (but that is a calculation
>Confluent will have to make).
>
>The other, independent, concern is that REST is something that is expected
>to be available out of the box with Kafka. I personally don't feel
>strongly
>about it (better use proper, efficient APIs from day one), though it is
>definitely way smaller than adding a stream processing engine to the
>project :)
>Again,the kafka-rest "community" could take steps to make it even easier
>to
>install, configure and run kafka-rest for new users on vanilla Apache
>Kafka
>(outside the Confluent platform), if they wish that (or welcome
>c

Re: [DISCUSS] KIP-80: Kafka REST Server

[Suresh Srinivas]

ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.

As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.

Regards,
Suresh

On 10/6/16, 8:32 AM, "Ofir Manor"  wrote:

>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka  community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that, existing REST API project doesn't support new client API
>and
>>> hence there is no security support either.
>
>This might be easy to fix. Maybe Confluent / kafka-rest community can
>clarify that - what is their contribution policy, dev style, roadmap etc.
>If they want, they can make an effort to encourage participation from
>people outside Confluent (easily accept contributions, invite external
>commiters or have open dev process similar to Apache Kafka etc), as there
>is definitely seems to be some interest on the list. That might clear the
>community concern and help kafka-rest project (but that is a calculation
>Confluent will have to make).
>
>The other, independent, concern is that REST is something that is expected
>to be available out of the box with Kafka. I personally don't feel
>strongly
>about it (better use proper, efficient APIs from day one), though it is
>definitely way smaller than adding a stream processing engine to the
>project :)
>Again,the kafka-rest "community" could take steps to make it even easier
>to
>install, configure and run kafka-rest for new users on vanilla Apache
>Kafka
>(outside the Confluent platform), if they wish that (or welcome
>contributions to that end), but that is up to them.
>Finally, if after the above steps were taken there would still a strong
>desire to include a great rest gateway with Apache Kafka, I assume the
>community could hypothetically fork the existing kafka-rest into an Apache
>Kafka subproject and maintain it "within Apache" instead of implementing
>it
>from scratch (though I'm not a lawyer etc) - but I cannot imagine it
>happen
>without Confluent blessing, and I think that is likely much less optimal
>(pulling in other Confluent / Apache licensed dependencies) than having a
>separate external community around kafka-rest.
>
>
>Just my two cents,
>
>
>Ofir Manor
>
>Co-Founder & CTO | Equalum
>
>Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
>On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani 
>wrote:
>
>> Neha & Jay,
>>  We did look at the open source alternatives. Our
>>concern
>> is what's the patch acceptance and adding features/ bug-fixes to the
>> existing project under a Github (although it's licensed under Apache
>>2.0).
>> It would be great if that project made available under Apache and
>>driven by
>> the community.  Adding to the above, not all Kafka users are interested
>>in
>> using the Java client API, they would like to have simple REST API where
>> they can code against using any language. I do believe this adds value
>>to
>> Apache Kafka in itself.
>>
>> "For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> reinventing the wheel. What I'm looking for is a detailed comparison of
>>the
>> gaps and why those can't be improved in the REST proxy that already
>>exists
>> and is actively maintained."
>>
>> We are not looking at this as  NIH. What we are worried about is a
>>project
>> that's not maintained in a community. So the process of accepting
>>patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
>>and
>> hence there is no security support either.
>> We don't know the timeline when that's made available. We would like to
>>add
>> admin functionality into the REST API. So the Roadmap of 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Neha Narkhede]

Harsha/Mani,

I completely agree that adding admin API support and security are important
features for the Kafka REST proxy. Luckily the roadmap items that you
mentioned as being important for a Kafka REST proxy server are exactly the
ones the community working on this REST proxy want to add to it :-)

But I haven't seen any community emails or patches being submitted by you
guys, so I'm wondering why you are concerned about whether the community is
open to accepting patches or not.

Does that make sense?

On Fri, Oct 7, 2016 at 12:03 PM Harsha Chintalapani  wrote:

> Ofir,
> …
> " personally think it would be quite wasteful to re-implement the REST
> gateway just because that an actively-maintained piece of Apache-licensed
> software is not governed directly by the Apache Kafka community... While
> kafka-rest repo is owned by Confluent, the contributors including the main
> one are also part of the Apache Kafka  community, so there is a chance to
> work this out."
> It is important for the code to be under Apache guidelines for
> contributions so that it is driven by the community.
> As part of Kafka community, we would like to improve the project and add
> new features. If kafka-rest repo can be donated to Apache Kafka project
> we will be more than happy to welcome that and add our features on top of
> it.
>
> Thanks,
> harsha
>
> On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps  wrote:
>
> > Hi Manikumar,
> >
> > I agree totally agree that REST is important. What I don't understand is
> > why we'd duplicate the existing REST interface inside the Kafka project.
> > That seems to needlessly fragment things.
> >
> > -Jay
> >
> > On Sat, Oct 1, 2016 at 5:38 AM, Manikumar 
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Thanks for your reply.
> > >
> > > I agree that we can not add all the clients/tools available in
> ecosystem
> > > page to Kafka repo itself. But we feel REST Interface is different from
> > > other clients/tools. Since any language that can work with HTTP can
> > > easily integrate with this interface, Having an "official"  REST
> > > interface helps user community. This also helps us to integrate well
> > > with external management and provisioning tools.  Apache Kafka release
> > > with Java clients + REST interface is sufficient for most of the user
> > > deployments/requirements. This helps users to deal with less number
> > > of distributions/builds.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
> > >
> > > > Hey guys,
> > > >
> > > > There's already a REST interface maintained as a separate
> project--it's
> > > > open source and apache licensed and actively maintained (
> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> that?
> > > You
> > > > mentioned that there was some compatibility concern, but
> compatibility
> > > has
> > > > to do with the consumer protocol guarantees not the repo the code is
> > in,
> > > > right? Not sure that concern makes sense.
> > > >
> > > > We could argue for adding pretty much anything and everything in the
> > > > ecosystem page in Kafka itself but I'm not sure that would make the
> > > project
> > > > more agile.
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> manikumar.re...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > >
> > > > > There are already open-source alternatives are available.  But we
> > would
> > > > > like to add REST server that
> > > > > many users ask for under Apache Kafka repo. Many data Infra tools
> > comes
> > > > up
> > > > > with Rest Interface.
> > > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > > messages
> > > > > and admin interface for
> > > > > integrating with external management and provisioning tools.This
> will
> > > > also
> > > > > allow the maintenance of
> > > > > REST server and adding new features makes it easy because apache
> > > > community.
> > > > >
> > > > > The KIP wiki is the following:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 80%3A+Kafka+Rest+Server
> > > > >
> > > > > Your comments and feedback are welcome.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >
>
-- 
Thanks,
Neha

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Chintalapani]

Ofir,
…
" personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka  community, so there is a chance to
work this out."
It is important for the code to be under Apache guidelines for
contributions so that it is driven by the community.
As part of Kafka community, we would like to improve the project and add
new features. If kafka-rest repo can be donated to Apache Kafka project
we will be more than happy to welcome that and add our features on top of
it.

Thanks,
harsha

On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps  wrote:

> Hi Manikumar,
>
> I agree totally agree that REST is important. What I don't understand is
> why we'd duplicate the existing REST interface inside the Kafka project.
> That seems to needlessly fragment things.
>
> -Jay
>
> On Sat, Oct 1, 2016 at 5:38 AM, Manikumar 
> wrote:
>
> > Hi Jay,
> >
> > Thanks for your reply.
> >
> > I agree that we can not add all the clients/tools available in ecosystem
> > page to Kafka repo itself. But we feel REST Interface is different from
> > other clients/tools. Since any language that can work with HTTP can
> > easily integrate with this interface, Having an "official"  REST
> > interface helps user community. This also helps us to integrate well
> > with external management and provisioning tools.  Apache Kafka release
> > with Java clients + REST interface is sufficient for most of the user
> > deployments/requirements. This helps users to deal with less number
> > of distributions/builds.
> >
> > Thanks,
> > Manikumar
> >
> >
> > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
> >
> > > Hey guys,
> > >
> > > There's already a REST interface maintained as a separate project--it's
> > > open source and apache licensed and actively maintained (
> > > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> > You
> > > mentioned that there was some compatibility concern, but compatibility
> > has
> > > to do with the consumer protocol guarantees not the repo the code is
> in,
> > > right? Not sure that concern makes sense.
> > >
> > > We could argue for adding pretty much anything and everything in the
> > > ecosystem page in Kafka itself but I'm not sure that would make the
> > project
> > > more agile.
> > >
> > > -Jay
> > >
> > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar  >
> > > wrote:
> > >
> > > > Hi Kafka Devs,
> > > >
> > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > >
> > > > There are already open-source alternatives are available.  But we
> would
> > > > like to add REST server that
> > > > many users ask for under Apache Kafka repo. Many data Infra tools
> comes
> > > up
> > > > with Rest Interface.
> > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > messages
> > > > and admin interface for
> > > > integrating with external management and provisioning tools.This will
> > > also
> > > > allow the maintenance of
> > > > REST server and adding new features makes it easy because apache
> > > community.
> > > >
> > > > The KIP wiki is the following:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 80%3A+Kafka+Rest+Server
> > > >
> > > > Your comments and feedback are welcome.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Ch]

Ofir,
" personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka  community, so there is a chance to
work this out."
It is important for the code to be under Apache guidelines for
contributions so that it is driven by the community.
As part of Kafka community, we would like to improve the project and add
new features. If kafka-rest repo can be donated to Apache Kafka project
we will be more than happy to welcome that and add our features on top of
it.

Thanks,
harsha

On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps  wrote:

Hi Manikumar,

I agree totally agree that REST is important. What I don't understand is
why we'd duplicate the existing REST interface inside the Kafka project.
That seems to needlessly fragment things.

-Jay

On Sat, Oct 1, 2016 at 5:38 AM, Manikumar  wrote:

> Hi Jay,
>
> Thanks for your reply.
>
> I agree that we can not add all the clients/tools available in ecosystem
> page to Kafka repo itself. But we feel REST Interface is different from
> other clients/tools. Since any language that can work with HTTP can
> easily integrate with this interface, Having an "official"  REST
> interface helps user community. This also helps us to integrate well
> with external management and provisioning tools.  Apache Kafka release
> with Java clients + REST interface is sufficient for most of the user
> deployments/requirements. This helps users to deal with less number
> of distributions/builds.
>
> Thanks,
> Manikumar
>
>
> On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
>
> > Hey guys,
> >
> > There's already a REST interface maintained as a separate project--it's
> > open source and apache licensed and actively maintained (
> > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> You
> > mentioned that there was some compatibility concern, but compatibility
> has
> > to do with the consumer protocol guarantees not the repo the code is in,
> > right? Not sure that concern makes sense.
> >
> > We could argue for adding pretty much anything and everything in the
> > ecosystem page in Kafka itself but I'm not sure that would make the
> project
> > more agile.
> >
> > -Jay
> >
> > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar 
> > wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >
> > > There are already open-source alternatives are available.  But we
would
> > > like to add REST server that
> > > many users ask for under Apache Kafka repo. Many data Infra tools
comes
> > up
> > > with Rest Interface.
> > > It is useful to have inbuilt Rest API support for Produce, Consume
> > messages
> > > and admin interface for
> > > integrating with external management and provisioning tools.This will
> > also
> > > allow the maintenance of
> > > REST server and adding new features makes it easy because apache
> > community.
> > >
> > > The KIP wiki is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 80%3A+Kafka+Rest+Server
> > >
> > > Your comments and feedback are welcome.
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>

Re: [DISCUSS] KIP-80: Kafka REST Server

[Jay Kreps]

Hi Manikumar,

I agree totally agree that REST is important. What I don't understand is
why we'd duplicate the existing REST interface inside the Kafka project.
That seems to needlessly fragment things.

-Jay

On Sat, Oct 1, 2016 at 5:38 AM, Manikumar  wrote:

> Hi Jay,
>
> Thanks for your reply.
>
> I agree that we can not add all the clients/tools available in ecosystem
> page to Kafka repo itself. But we feel REST Interface is different from
> other clients/tools. Since any language that can work with HTTP can
> easily integrate with this interface, Having an "official"  REST
> interface helps user community. This also helps us to integrate well
> with external management and provisioning tools.  Apache Kafka release
> with Java clients + REST interface is sufficient for most of the user
> deployments/requirements. This helps users to deal with less number
> of distributions/builds.
>
> Thanks,
> Manikumar
>
>
> On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
>
> > Hey guys,
> >
> > There's already a REST interface maintained as a separate project--it's
> > open source and apache licensed and actively maintained (
> > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> You
> > mentioned that there was some compatibility concern, but compatibility
> has
> > to do with the consumer protocol guarantees not the repo the code is in,
> > right? Not sure that concern makes sense.
> >
> > We could argue for adding pretty much anything and everything in the
> > ecosystem page in Kafka itself but I'm not sure that would make the
> project
> > more agile.
> >
> > -Jay
> >
> > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar 
> > wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >
> > > There are already open-source alternatives are available.  But we would
> > > like to add REST server that
> > > many users ask for under Apache Kafka repo. Many data Infra tools comes
> > up
> > > with Rest Interface.
> > > It is useful to have inbuilt Rest API support for Produce, Consume
> > messages
> > > and admin interface for
> > > integrating with external management and provisioning tools.This will
> > also
> > > allow the maintenance of
> > > REST server and adding new features makes it easy because apache
> > community.
> > >
> > > The KIP wiki is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 80%3A+Kafka+Rest+Server
> > >
> > > Your comments and feedback are welcome.
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ofir Manor]

I personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka  community, so there is a chance to
work this out.

However, there are two valid concerns here that could be addressed, around
community and accessibility:
>> What we are worried about is a project
>> that's not maintained in a community. So the process of accepting patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
and
>> hence there is no security support either.

This might be easy to fix. Maybe Confluent / kafka-rest community can
clarify that - what is their contribution policy, dev style, roadmap etc.
If they want, they can make an effort to encourage participation from
people outside Confluent (easily accept contributions, invite external
commiters or have open dev process similar to Apache Kafka etc), as there
is definitely seems to be some interest on the list. That might clear the
community concern and help kafka-rest project (but that is a calculation
Confluent will have to make).

The other, independent, concern is that REST is something that is expected
to be available out of the box with Kafka. I personally don't feel strongly
about it (better use proper, efficient APIs from day one), though it is
definitely way smaller than adding a stream processing engine to the
project :)
Again,the kafka-rest "community" could take steps to make it even easier to
install, configure and run kafka-rest for new users on vanilla Apache Kafka
(outside the Confluent platform), if they wish that (or welcome
contributions to that end), but that is up to them.
Finally, if after the above steps were taken there would still a strong
desire to include a great rest gateway with Apache Kafka, I assume the
community could hypothetically fork the existing kafka-rest into an Apache
Kafka subproject and maintain it "within Apache" instead of implementing it
from scratch (though I'm not a lawyer etc) - but I cannot imagine it happen
without Confluent blessing, and I think that is likely much less optimal
(pulling in other Confluent / Apache licensed dependencies) than having a
separate external community around kafka-rest.


Just my two cents,


Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani 
wrote:

> Neha & Jay,
>  We did look at the open source alternatives. Our concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache 2.0).
> It would be great if that project made available under Apache and driven by
> the community.  Adding to the above, not all Kafka users are interested in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value to
> Apache Kafka in itself.
>
> "For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained."
>
> We are not looking at this as  NIH. What we are worried about is a project
> that's not maintained in a community. So the process of accepting patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
>
>
> "This doesn't materially have an impact on expanding the usability of
>Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
>Kafka users, and maybe 10% of those are the ones who will use the REST
>proxy. The remaining 50% are non-java client users (C, python, go, node
>etc)."
>
> REST API is most often asked feature in my interactions with Kafka users.
> In an organization, There will be independent teams who will write their
>  Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and more users
> start using Kafka.
>
> "More surface area means more work to keep things consistent. Failure
>to do that has, in fact, hurt the user experience."
> Having myriad Kafka client GitHub 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Ben Davison]

I also think it would be good to have a REST client that can interact with
the new management API's coming down the pipe.

On Tue, Oct 4, 2016 at 10:35 PM, Edoardo Comar <eco...@uk.ibm.com> wrote:

> Harsha
> thanks for opening the discussion on this KIP.
>
> While I understand he founding members' stand that the Kafka project can
> not expand its surface to a large number of clients,
> I strongly agree with your well explained points below and support your
> KIP.
>
> A REST API is not just on the same level as any client, it's a basic
> building block of open web technologies.
> It's the API that most of our first time user want to try out (or that
> would be users ask for and expect to be there).
>
> A REST API for Kafka and a robust server implementation
> under the open governance of the Apache community would be most welcome.
>
> +1
>
> Edo
> --
> Edoardo Comar
> IBM MessageHub
>
> IBM United Kingdom Limited Registered in England and Wales with number
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
>
> Harsha Chintalapani <ka...@harsha.io> wrote on 02/10/2016 21:23:15:
>
> > From: Harsha Chintalapani <ka...@harsha.io>
> > To: dev@kafka.apache.org
> > Date: 02/10/2016 21:23
> > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> >
> > Neha & Jay,
> >  We did look at the open source alternatives. Our
> concern
> > is what's the patch acceptance and adding features/ bug-fixes to the
> > existing project under a Github (although it's licensed under Apache
> 2.0).
> > It would be great if that project made available under Apache and driven
> by
> > the community.  Adding to the above, not all Kafka users are interested
> in
> > using the Java client API, they would like to have simple REST API where
> > they can code against using any language. I do believe this adds value
> to
> > Apache Kafka in itself.
> >
> > "For 1, I don't think there is value in giving in to the NIH syndrome
> and
> > reinventing the wheel. What I'm looking for is a detailed comparison of
> the
> > gaps and why those can't be improved in the REST proxy that already
> exists
> > and is actively maintained."
> >
> > We are not looking at this as  NIH. What we are worried about is a
> project
> > that's not maintained in a community. So the process of accepting
> patches
> > and priorities is not clear, and it's not developed in Apache community.
> > Not only that, existing REST API project doesn't support new client API
> and
> > hence there is no security support either.
> > We don't know the timeline when that's made available. We would like to
> add
> > admin functionality into the REST API. So the Roadmap of that project is
> > not driven by Apache.
> >
> >
> > "This doesn't materially have an impact on expanding the usability of
> >Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
> >Kafka users, and maybe 10% of those are the ones who will use the
> REST
> >proxy. The remaining 50% are non-java client users (C, python, go,
> node
> >etc)."
> >
> > REST API is most often asked feature in my interactions with Kafka
> users.
> > In an organization, There will be independent teams who will write their
> >  Kafka clients using different language libraries available today, and
> > there is no way to standardize this. Instead of supporting several
> > different client libraries users will be interested in using a REST API
> > server. The need for a REST API will only increase as more and more
> users
> > start using Kafka.
> >
> > "More surface area means more work to keep things consistent. Failure
> >to do that has, in fact, hurt the user experience."
> > Having myriad Kafka client GitHub projects that support different
> languages
> > hurts the user experience and pushes burden to maintain these libraries.
> > REST API is a simple code base that uses existing java client libraries
> to
> > make life easier for the users.
> >
> > Thanks,
> > Harsha
> >
> > On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <n...@confluent.io> wrote:
> >
> > > Manikumar,
> > >
> > > Thanks for sharing the proposal. I think there are 2 parts to this
> > > discussion -
> > >
> > > 1. Should we rewrite a REST proxy given that there is a
> feature-complete,
> > > open-source and actively maintained REST proxy in the communi

Re: [DISCUSS] KIP-80: Kafka REST Server

[Edoardo Comar]

Harsha
thanks for opening the discussion on this KIP.

While I understand he founding members' stand that the Kafka project can 
not expand its surface to a large number of clients,
I strongly agree with your well explained points below and support your 
KIP.

A REST API is not just on the same level as any client, it's a basic 
building block of open web technologies.
It's the API that most of our first time user want to try out (or that 
would be users ask for and expect to be there).

A REST API for Kafka and a robust server implementation 
under the open governance of the Apache community would be most welcome.

+1

Edo
--
Edoardo Comar
IBM MessageHub

IBM United Kingdom Limited Registered in England and Wales with number 
741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 
3AU

Harsha Chintalapani <ka...@harsha.io> wrote on 02/10/2016 21:23:15:

> From: Harsha Chintalapani <ka...@harsha.io>
> To: dev@kafka.apache.org
> Date: 02/10/2016 21:23
> Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> 
> Neha & Jay,
>  We did look at the open source alternatives. Our 
concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache 
2.0).
> It would be great if that project made available under Apache and driven 
by
> the community.  Adding to the above, not all Kafka users are interested 
in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value 
to
> Apache Kafka in itself.
> 
> "For 1, I don't think there is value in giving in to the NIH syndrome 
and
> reinventing the wheel. What I'm looking for is a detailed comparison of 
the
> gaps and why those can't be improved in the REST proxy that already 
exists
> and is actively maintained."
> 
> We are not looking at this as  NIH. What we are worried about is a 
project
> that's not maintained in a community. So the process of accepting 
patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API 
and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to 
add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
> 
> 
> "This doesn't materially have an impact on expanding the usability of
>Kafka. In my experience, REST proxy + Java clients only cover ~50% of 
all
>Kafka users, and maybe 10% of those are the ones who will use the 
REST
>proxy. The remaining 50% are non-java client users (C, python, go, 
node
>etc)."
> 
> REST API is most often asked feature in my interactions with Kafka 
users.
> In an organization, There will be independent teams who will write their
>  Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and more 
users
> start using Kafka.
> 
> "More surface area means more work to keep things consistent. Failure
>to do that has, in fact, hurt the user experience."
> Having myriad Kafka client GitHub projects that support different 
languages
> hurts the user experience and pushes burden to maintain these libraries.
> REST API is a simple code base that uses existing java client libraries 
to
> make life easier for the users.
> 
> Thanks,
> Harsha
> 
> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <n...@confluent.io> wrote:
> 
> > Manikumar,
> >
> > Thanks for sharing the proposal. I think there are 2 parts to this
> > discussion -
> >
> > 1. Should we rewrite a REST proxy given that there is a 
feature-complete,
> > open-source and actively maintained REST proxy in the community?
> > 2. Does adding a REST proxy to Apache Kafka make us more agile and 
maintain
> > the high-quality experience that Kafka users have today?
> >
> > For 1, I don't think there is value in giving in to the NIH syndrome 
and
> > reinventing the wheel. What I'm looking for is a detailed comparison 
of the
> > gaps and why those can't be improved in the REST proxy that already 
exists
> > and is actively maintained. For example, we depend on zkClient and 
have
> > found as well as fixed several bugs by working closely with the people 
who
> > maintain zkClient. This should be possible for REST proxy as well, 
right?
> >
> > For 2, I'd l

Re: [DISCUSS] KIP-80: Kafka REST Server

[Manikumar]

Hi Jay,

Thanks for your reply.

I agree that we can not add all the clients/tools available in ecosystem
page to Kafka repo itself. But we feel REST Interface is different from
other clients/tools. Since any language that can work with HTTP can
easily integrate with this interface, Having an "official"  REST
interface helps user community. This also helps us to integrate well
with external management and provisioning tools.  Apache Kafka release
with Java clients + REST interface is sufficient for most of the user
deployments/requirements. This helps users to deal with less number
of distributions/builds.

Thanks,
Manikumar


On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:

> Hey guys,
>
> There's already a REST interface maintained as a separate project--it's
> open source and apache licensed and actively maintained (
> https://github.com/confluentinc/kafka-rest). What is wrong with that? You
> mentioned that there was some compatibility concern, but compatibility has
> to do with the consumer protocol guarantees not the repo the code is in,
> right? Not sure that concern makes sense.
>
> We could argue for adding pretty much anything and everything in the
> ecosystem page in Kafka itself but I'm not sure that would make the project
> more agile.
>
> -Jay
>
> On Wed, Sep 28, 2016 at 12:04 AM, Manikumar 
> wrote:
>
> > Hi Kafka Devs,
> >
> > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> >
> > There are already open-source alternatives are available.  But we would
> > like to add REST server that
> > many users ask for under Apache Kafka repo. Many data Infra tools comes
> up
> > with Rest Interface.
> > It is useful to have inbuilt Rest API support for Produce, Consume
> messages
> > and admin interface for
> > integrating with external management and provisioning tools.This will
> also
> > allow the maintenance of
> > REST server and adding new features makes it easy because apache
> community.
> >
> > The KIP wiki is the following:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 80%3A+Kafka+Rest+Server
> >
> > Your comments and feedback are welcome.
> >
> > Thanks,
> > Manikumar
> >
>