Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Gwen Shapira]

Jayesh, if you have a github repo with your tool, we can add it to the
ecosystem page in the wiki, so more people will find it.

On Aug 17, 2016 8:00 PM, "Jayesh Thakrar" <j_thak...@yahoo.com.invalid>
wrote:

> Thank you Grant and Gwen for the valuable feedback and suggestions.Yes, it
> makes sense to wait for the admin client API.In the meantime, I can spin
> this off as a standalone tool and allow things to bake, giving the utility
> more field usage and maturity.
> As for the endpoint info, I included it in the KIP because the information
> was available in the current API.I am sure once the admin API is released,
> the KIP can be revisited.
> Thanks for sharing your insights!
> Jayesh
>
>   From: Grant Henke <ghe...@cloudera.com>
>  To: dev@kafka.apache.org
> Cc: Jayesh Thakrar <j_thak...@yahoo.com>
>  Sent: Wednesday, August 17, 2016 9:17 PM
>  Subject: Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command -
> kafka-brokers.sh
>
> Hi Jayesh,
> Like Gwen said KIP-4 is adding fields and requests to the wire protocols
> in order to allow all admin tools to talk directly to Kafka and a client
> api to support those requests. Talking to Kafka as opposed to Zookeeper
> allows us to leverage authorization, lock down zookeeper, and improve
> compatibility. Like Gwen said waiting until some of the KIP-4 work is done
> may avoid rework. I can't make a commitment to how fast the client will be
> available as it depends on many factors but progress is being made
> regularly and I do have some mock client work done locally.
> It looks like the only thing in your proposal that can not be supported
> via the wire protocol today is the endpoints metadata. It's sort of a
> catch-22 because the bootstrap is required to connect to a Kafka cluster
> (as opposed to zookeeper) and at that point the Metadata returned assumes
> an endpoint of the connecting security protocol. Is that an acceptable
> limitation?
> Thanks,Grant
> On Wed, Aug 17, 2016 at 6:53 PM, Gwen Shapira <g...@confluent.io> wrote:
>
> Thanks Jayesh.
>
> I think this can be a useful addition to Apache Kafka.
> One potential issue is that you are getting all the information for
> ZooKeeper. We already added a protocol that allows adding the
> information to Kafka itself and are in the process of adding an admin
> client (i.e. Java client, not CLI).
> If you add this as planned, we'll need to re-work it to work with
> Kafka directly instead of ZooKeeper once the admin client lands. If
> you choose, you can wait for the admin client to arrive first, and
> avoid the re-work.
>
> Gwen
>
> On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
> <j_thak...@yahoo.com.invalid> wrote:
> > All,
> > If there is no discussion, feedback or objection, is there any concern
> in going to the next step of voting?
> > Thanks,Jayesh
> >   From: Jayesh Thakrar <j_thak...@yahoo.com>
> >  To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> >  Sent: Saturday, August 13, 2016 11:44 PM
> >  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command -
> kafka-brokers.sh
> >
> > This is to start off a discussion on the above KIP at
> > https://cwiki.apache.org/ confluence/display/KAFKA/KIP-
> 59%3A+Proposal+for+a+kafka+ broker+commandThe proposal is to fill the void
> of a command line tool/utility that can provide information on the brokers
> in a Kafka cluster.
> > The code is available on GitHub at https://github.com/JThakrar/
> kafkaThe KIP page has the help documentation as well as the output from the
> command with various options.Thank you,Jayesh Thakrar
> >
> >
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
>
>
> --
> Grant Henke
> Software Engineer | clouderagr...@cloudera.com | twitter.com/gchenke |
> linkedin.com/in/granthenke
>
>

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Jayesh Thakrar]

Thank you Grant and Gwen for the valuable feedback and suggestions.Yes, it 
makes sense to wait for the admin client API.In the meantime, I can spin this 
off as a standalone tool and allow things to bake, giving the utility more 
field usage and maturity.
As for the endpoint info, I included it in the KIP because the information was 
available in the current API.I am sure once the admin API is released, the KIP 
can be revisited.
Thanks for sharing your insights!
Jayesh 

  From: Grant Henke <ghe...@cloudera.com>
 To: dev@kafka.apache.org 
Cc: Jayesh Thakrar <j_thak...@yahoo.com>
 Sent: Wednesday, August 17, 2016 9:17 PM
 Subject: Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
kafka-brokers.sh
   
Hi Jayesh,
Like Gwen said KIP-4 is adding fields and requests to the wire protocols in 
order to allow all admin tools to talk directly to Kafka and a client api to 
support those requests. Talking to Kafka as opposed to Zookeeper allows us to 
leverage authorization, lock down zookeeper, and improve compatibility. Like 
Gwen said waiting until some of the KIP-4 work is done may avoid rework. I 
can't make a commitment to how fast the client will be available as it depends 
on many factors but progress is being made regularly and I do have some mock 
client work done locally. 
It looks like the only thing in your proposal that can not be supported via the 
wire protocol today is the endpoints metadata. It's sort of a catch-22 because 
the bootstrap is required to connect to a Kafka cluster (as opposed to 
zookeeper) and at that point the Metadata returned assumes an endpoint of the 
connecting security protocol. Is that an acceptable limitation?
Thanks,Grant
On Wed, Aug 17, 2016 at 6:53 PM, Gwen Shapira <g...@confluent.io> wrote:

Thanks Jayesh.

I think this can be a useful addition to Apache Kafka.
One potential issue is that you are getting all the information for
ZooKeeper. We already added a protocol that allows adding the
information to Kafka itself and are in the process of adding an admin
client (i.e. Java client, not CLI).
If you add this as planned, we'll need to re-work it to work with
Kafka directly instead of ZooKeeper once the admin client lands. If
you choose, you can wait for the admin client to arrive first, and
avoid the re-work.

Gwen

On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
<j_thak...@yahoo.com.invalid> wrote:
> All,
> If there is no discussion, feedback or objection, is there any concern in 
> going to the next step of voting?
> Thanks,Jayesh
>       From: Jayesh Thakrar <j_thak...@yahoo.com>
>  To: "dev@kafka.apache.org" <dev@kafka.apache.org>
>  Sent: Saturday, August 13, 2016 11:44 PM
>  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
>kafka-brokers.sh
>
> This is to start off a discussion on the above KIP at
> https://cwiki.apache.org/ confluence/display/KAFKA/KIP- 
> 59%3A+Proposal+for+a+kafka+ broker+commandThe proposal is to fill the void of 
> a command line tool/utility that can provide information on the brokers in a 
> Kafka cluster.
> The code is available on GitHub at https://github.com/JThakrar/ kafkaThe KIP 
> page has the help documentation as well as the output from the command with 
> various options.Thank you,Jayesh Thakrar
>
>
>



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog




-- 
Grant Henke 
Software Engineer | clouderagr...@cloudera.com | twitter.com/gchenke | 
linkedin.com/in/granthenke

  

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Grant Henke]

Hi Jayesh,

Like Gwen said KIP-4 is adding fields and requests to the wire protocols in
order to allow all admin tools to talk directly to Kafka and a client api
to support those requests. Talking to Kafka as opposed to Zookeeper allows
us to leverage authorization, lock down zookeeper, and improve
compatibility. Like Gwen said waiting until some of the KIP-4 work is done
may avoid rework. I can't make a commitment to how fast the client will be
available as it depends on many factors but progress is being made
regularly and I do have some mock client work done locally.

It looks like the only thing in your proposal that can not be supported via
the wire protocol today is the endpoints metadata. It's sort of a catch-22
because the bootstrap is required to connect to a Kafka cluster (as opposed
to zookeeper) and at that point the Metadata returned assumes an endpoint
of the connecting security protocol. Is that an acceptable limitation?

Thanks,
Grant

On Wed, Aug 17, 2016 at 6:53 PM, Gwen Shapira <g...@confluent.io> wrote:

> Thanks Jayesh.
>
> I think this can be a useful addition to Apache Kafka.
> One potential issue is that you are getting all the information for
> ZooKeeper. We already added a protocol that allows adding the
> information to Kafka itself and are in the process of adding an admin
> client (i.e. Java client, not CLI).
> If you add this as planned, we'll need to re-work it to work with
> Kafka directly instead of ZooKeeper once the admin client lands. If
> you choose, you can wait for the admin client to arrive first, and
> avoid the re-work.
>
> Gwen
>
> On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
> <j_thak...@yahoo.com.invalid> wrote:
> > All,
> > If there is no discussion, feedback or objection, is there any concern
> in going to the next step of voting?
> > Thanks,Jayesh
> >   From: Jayesh Thakrar <j_thak...@yahoo.com>
> >  To: "dev@kafka.apache.org" <dev@kafka.apache.org>
> >  Sent: Saturday, August 13, 2016 11:44 PM
> >  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command -
> kafka-brokers.sh
> >
> > This is to start off a discussion on the above KIP at
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 59%3A+Proposal+for+a+kafka+broker+commandThe proposal is to fill the void
> of a command line tool/utility that can provide information on the brokers
> in a Kafka cluster.
> > The code is available on GitHub at https://github.com/JThakrar/kafkaThe
> KIP page has the help documentation as well as the output from the command
> with various options.Thank you,Jayesh Thakrar
> >
> >
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Gwen Shapira]

Thanks Jayesh.

I think this can be a useful addition to Apache Kafka.
One potential issue is that you are getting all the information for
ZooKeeper. We already added a protocol that allows adding the
information to Kafka itself and are in the process of adding an admin
client (i.e. Java client, not CLI).
If you add this as planned, we'll need to re-work it to work with
Kafka directly instead of ZooKeeper once the admin client lands. If
you choose, you can wait for the admin client to arrive first, and
avoid the re-work.

Gwen

On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
<j_thak...@yahoo.com.invalid> wrote:
> All,
> If there is no discussion, feedback or objection, is there any concern in 
> going to the next step of voting?
> Thanks,Jayesh
>   From: Jayesh Thakrar <j_thak...@yahoo.com>
>  To: "dev@kafka.apache.org" <dev@kafka.apache.org>
>  Sent: Saturday, August 13, 2016 11:44 PM
>  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
> kafka-brokers.sh
>
> This is to start off a discussion on the above KIP at
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+commandThe
>  proposal is to fill the void of a command line tool/utility that can provide 
> information on the brokers in a Kafka cluster.
> The code is available on GitHub at https://github.com/JThakrar/kafkaThe KIP 
> page has the help documentation as well as the output from the command with 
> various options.Thank you,Jayesh Thakrar
>
>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Jayesh Thakrar]

All,
If there is no discussion, feedback or objection, is there any concern in going 
to the next step of voting?
Thanks,Jayesh
  From: Jayesh Thakrar <j_thak...@yahoo.com>
 To: "dev@kafka.apache.org" <dev@kafka.apache.org> 
 Sent: Saturday, August 13, 2016 11:44 PM
 Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
kafka-brokers.sh
   
This is to start off a discussion on the above KIP at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+commandThe
 proposal is to fill the void of a command line tool/utility that can provide 
information on the brokers in a Kafka cluster.
The code is available on GitHub at https://github.com/JThakrar/kafkaThe KIP 
page has the help documentation as well as the output from the command with 
various options.Thank you,Jayesh Thakrar


   

[DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Jayesh Thakrar]

This is to start off a discussion on the above KIP at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+commandThe
 proposal is to fill the void of a command line tool/utility that can provide 
information on the brokers in a Kafka cluster.
The code is available on GitHub at https://github.com/JThakrar/kafkaThe KIP 
page has the help documentation as well as the output from the command with 
various options.Thank you,Jayesh Thakrar

RE: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Thakrar, Jayesh]

Thanks Gwen - yes, I agree - let me work on it, make it available on github and 
then I guess we can go from there.

Thanks,
Jayesh


-Original Message-
From: Gwen Shapira [mailto:g...@confluent.io] 
Sent: Wednesday, May 11, 2016 12:26 PM
To: dev@kafka.apache.org; Jayesh Thakrar <j_thak...@yahoo.com>
Cc: Users <us...@kafka.apache.org>
Subject: Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
kafka-brokers.sh

Hello Jayesh,

Thank you for the suggestion. I like the proposal and the new tool seems useful.

Do you already have the tool available in a github repository?

If you don't, then this would be a good place to start - there are many Kafka 
utilities in github repositories (Yahoo's Kafka Manager as a famous example). 
This way users can evaluate the usefulness of the new tool before we include it 
in core-kafka.

Gwen

On Mon, May 9, 2016 at 8:50 PM, Jayesh Thakrar <j_thak...@yahoo.com.invalid> 
wrote:
> Hi All,
>
>
>
> This is to start off a discussion on the above KIP at 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+f
> or+a+kafka+broker+command The proposal is to fill the void of a 
> command line tool/utility that can provide information on the cluster and 
> brokers in a Kafka cluster.
> Thank you,Jayesh Thakrar
>
>





This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Gwen Shapira]

Hello Jayesh,

Thank you for the suggestion. I like the proposal and the new tool seems useful.

Do you already have the tool available in a github repository?

If you don't, then this would be a good place to start - there are
many Kafka utilities in github repositories (Yahoo's Kafka Manager as
a famous example). This way users can evaluate the usefulness of the
new tool before we include it in core-kafka.

Gwen

On Mon, May 9, 2016 at 8:50 PM, Jayesh Thakrar
 wrote:
> Hi All,
>
>
>
> This is to start off a discussion on the above KIP at 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
> The proposal is to fill the void of a command line tool/utility that can 
> provide information on the cluster and brokers in a Kafka cluster.
> Thank you,Jayesh Thakrar
>
>

[DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Jayesh Thakrar]

Hi All,



This is to start off a discussion on the above KIP at 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
The proposal is to fill the void of a command line tool/utility that can 
provide information on the cluster and brokers in a Kafka cluster.
Thank you,Jayesh Thakrar

  

[DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

[Jayesh Thakrar]

Hi All,
This is to start off a discussion on the above KIP at 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59+-+Proposal+for+a+kafka+broker+command+-+kafka-brokers.sh
The proposal is to fill the void of a command line tool/utility that can 
provide information on the cluster and brokers in a Kafka cluster.
Thank you,Jayesh Thakrar