Question: Lazy client connections and status check

2022-02-03 Thread Jayesh Thakrar
Hi All,

The admin, consumer and producer client creation are all lazy connections, in 
the sense that one only knows that the connection is “healthy” when an “action” 
is carried out (e.g., send a message).

There is no API call to check if the connection is healthy or if there are any 
issues with connectivity.
E.g., I could give an incorrect host/port address and will not know until I try 
the first action.

However, I do see a need for healthcheck after creating the client.
E.g., I could be looking at sending “yesterday’s data change for an 
application” which may be quite expensive. If I run the 1-hr query and then 
find that the Kafka connection was invalid, it’s a lot of time/effort wasted.

For such and other cases, it’s worthwhile to have a healthcheck that can be 
done before (and even after) running the expensive operation to validate the 
client connection.

Does it make sense to extend the client APIs (interfaces) with a healthcheck? 
Again, the term “healthcheck” may sound “sysadmin”, but it’s not. 
Alternatively, it may be even made implicit like in a JDBC connection, which if 
we know has been established assures that the client connection is ready for 
service.

Thanks,
Jayesh Thakrar





Disclaimer The information in this email and any attachments may contain 
proprietary and confidential information that is intended for the addressee(s) 
only. If you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution, retention or use of the contents of this 
information is prohibited. When addressed to our clients or vendors, any 
information contained in this e-mail or any attachments is subject to the terms 
and conditions in any governing contract. If you have received this e-mail in 
error, please immediately contact the sender and delete the e-mail.


[DISCUSS] KIP-59 : Proposal for a kafka broker command

2017-10-15 Thread Jayesh Thakrar
Hi All,
A wiki page and development for this KIP was completed last June.However, it 
utilized the older Admin API that used zookeeper.
Then I was told to hold-off until the new Admin API was ready.So this KIP lay 
dormant since then.I have rewritten the broker command now and would like to 
bring it up for discussion or know the next step if its not discussion.
Here are the details:
KIP Wiki: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
Jira: https://issues.apache.org/jira/browse/KAFKA-3663
Pull Request: https://github.com/apache/kafka/pull/1539
Thank you in advance for your help/guidance.
Jayesh Thakrar


[jira] [Commented] (KAFKA-4782) change chroot for a kafka instance

2017-02-20 Thread Jayesh Thakrar (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874650#comment-15874650
 ] 

Jayesh Thakrar commented on KAFKA-4782:
---

A good question and not an uncommon situation.
However I don't know if there is a generic tool/api to "move" a znode or set of 
znodes around.
I think you may have to write a custom program to do the task.
If you do, it could be a good contribution back to the Zookeeper community that 
will be useful to many beyond just the Kafka project.

> change chroot for a kafka instance
> --
>
> Key: KAFKA-4782
> URL: https://issues.apache.org/jira/browse/KAFKA-4782
> Project: Kafka
>  Issue Type: Improvement
>Reporter: polarbear
>
> Hello
> I am a beginner of the kafka.
> Our kafka is "kafka_2.11-0.10.0.1.jar.asc", should be 0.10.0.1.
> chroot is default "/".
> Our user ask us to change the chroot from "/" to "/kafka".
> I change the parameter zookeeper.connect to add /kafka, but we cannot find  
> the old topic in "/kafka", so I think it should be a new instance.
> Which tool can we use to migrate the topic from the old "/" to new "/kafka"?
> Thank you



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

2016-08-17 Thread Jayesh Thakrar
Thank you Grant and Gwen for the valuable feedback and suggestions.Yes, it 
makes sense to wait for the admin client API.In the meantime, I can spin this 
off as a standalone tool and allow things to bake, giving the utility more 
field usage and maturity.
As for the endpoint info, I included it in the KIP because the information was 
available in the current API.I am sure once the admin API is released, the KIP 
can be revisited.
Thanks for sharing your insights!
Jayesh 

  From: Grant Henke <ghe...@cloudera.com>
 To: dev@kafka.apache.org 
Cc: Jayesh Thakrar <j_thak...@yahoo.com>
 Sent: Wednesday, August 17, 2016 9:17 PM
 Subject: Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
kafka-brokers.sh
   
Hi Jayesh,
Like Gwen said KIP-4 is adding fields and requests to the wire protocols in 
order to allow all admin tools to talk directly to Kafka and a client api to 
support those requests. Talking to Kafka as opposed to Zookeeper allows us to 
leverage authorization, lock down zookeeper, and improve compatibility. Like 
Gwen said waiting until some of the KIP-4 work is done may avoid rework. I 
can't make a commitment to how fast the client will be available as it depends 
on many factors but progress is being made regularly and I do have some mock 
client work done locally. 
It looks like the only thing in your proposal that can not be supported via the 
wire protocol today is the endpoints metadata. It's sort of a catch-22 because 
the bootstrap is required to connect to a Kafka cluster (as opposed to 
zookeeper) and at that point the Metadata returned assumes an endpoint of the 
connecting security protocol. Is that an acceptable limitation?
Thanks,Grant
On Wed, Aug 17, 2016 at 6:53 PM, Gwen Shapira <g...@confluent.io> wrote:

Thanks Jayesh.

I think this can be a useful addition to Apache Kafka.
One potential issue is that you are getting all the information for
ZooKeeper. We already added a protocol that allows adding the
information to Kafka itself and are in the process of adding an admin
client (i.e. Java client, not CLI).
If you add this as planned, we'll need to re-work it to work with
Kafka directly instead of ZooKeeper once the admin client lands. If
you choose, you can wait for the admin client to arrive first, and
avoid the re-work.

Gwen

On Tue, Aug 16, 2016 at 7:22 AM, Jayesh Thakrar
<j_thak...@yahoo.com.invalid> wrote:
> All,
> If there is no discussion, feedback or objection, is there any concern in 
> going to the next step of voting?
> Thanks,Jayesh
>       From: Jayesh Thakrar <j_thak...@yahoo.com>
>  To: "dev@kafka.apache.org" <dev@kafka.apache.org>
>  Sent: Saturday, August 13, 2016 11:44 PM
>  Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
>kafka-brokers.sh
>
> This is to start off a discussion on the above KIP at
> https://cwiki.apache.org/ confluence/display/KAFKA/KIP- 
> 59%3A+Proposal+for+a+kafka+ broker+commandThe proposal is to fill the void of 
> a command line tool/utility that can provide information on the brokers in a 
> Kafka cluster.
> The code is available on GitHub at https://github.com/JThakrar/ kafkaThe KIP 
> page has the help documentation as well as the output from the command with 
> various options.Thank you,Jayesh Thakrar
>
>
>



--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog




-- 
Grant Henke 
Software Engineer | clouderagr...@cloudera.com | twitter.com/gchenke | 
linkedin.com/in/granthenke

  

Re: [DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

2016-08-16 Thread Jayesh Thakrar
All,
If there is no discussion, feedback or objection, is there any concern in going 
to the next step of voting?
Thanks,Jayesh
  From: Jayesh Thakrar <j_thak...@yahoo.com>
 To: "dev@kafka.apache.org" <dev@kafka.apache.org> 
 Sent: Saturday, August 13, 2016 11:44 PM
 Subject: [DISCUSS] KIP-59 - Proposal for a kafka broker command - 
kafka-brokers.sh
   
This is to start off a discussion on the above KIP at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+commandThe
 proposal is to fill the void of a command line tool/utility that can provide 
information on the brokers in a Kafka cluster.
The code is available on GitHub at https://github.com/JThakrar/kafkaThe KIP 
page has the help documentation as well as the output from the command with 
various options.Thank you,Jayesh Thakrar


   

[DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

2016-08-13 Thread Jayesh Thakrar
This is to start off a discussion on the above KIP at
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+commandThe
 proposal is to fill the void of a command line tool/utility that can provide 
information on the brokers in a Kafka cluster.
The code is available on GitHub at https://github.com/JThakrar/kafkaThe KIP 
page has the help documentation as well as the output from the command with 
various options.Thank you,Jayesh Thakrar


Re: How to get pull request reviewed - Pull request # 1539

2016-08-13 Thread Jayesh Thakrar
Hi Gwen,
Thanks for the reply.
I will initiate another discussion thread also.
Here's the earlier thread - 
https://mail-archives.apache.org/mod_mbox/kafka-users/201605.mbox/%3c545407a926534b4fbf3a62628c725a4cdcb5c...@ord-exdb102.corp.valueclick.com%3E
I guess I jumped the gun and did the discussion thread before the coding work 
:-)
Was thinking of getting the specs first before doing development work.
Thanks,Jayesh

  From: Gwen Shapira <g...@confluent.io>
 To: dev@kafka.apache.org; Jayesh Thakrar <j_thak...@yahoo.com> 
 Sent: Friday, August 12, 2016 9:40 PM
 Subject: Re: How to get pull request reviewed - Pull request # 1539
   
Hey,

In the case of your specific PR, there is also a KIP. Which means we
go by the KIP process:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

It looks like you followed steps #1 and #2 in the process (create the
wiki), now its time for step #3 - start a discussion thread.
The PR can only be reviewed and merged after the KIP vote passes.

Gwen

On Fri, Aug 12, 2016 at 8:48 AM, Jayesh Thakrar
<j_thak...@yahoo.com.invalid> wrote:
> Hi All,
> I am new to the Kafka community group development and was wondering how do I 
> get my pull request reviewed/processed?
> https://github.com/apache/kafka/pull/1539
>
> Am I missing any steps or is it just the backlog?
> Thanks,Jayesh



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog


  

How to get pull request reviewed - Pull request # 1539

2016-08-12 Thread Jayesh Thakrar
Hi All,
I am new to the Kafka community group development and was wondering how do I 
get my pull request reviewed/processed?
https://github.com/apache/kafka/pull/1539

Am I missing any steps or is it just the backlog?
Thanks,Jayesh


[jira] [Updated] (KAFKA-3663) Proposal for a kafka broker command - kafka-brokers.sh

2016-07-03 Thread Jayesh Thakrar (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh Thakrar updated KAFKA-3663:
--
Reviewer: Gwen Shapira

Added [~gwenshap] as reviewer.

> Proposal for a kafka broker command - kafka-brokers.sh
> --
>
> Key: KAFKA-3663
> URL: https://issues.apache.org/jira/browse/KAFKA-3663
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>        Reporter: Jayesh Thakrar
>
> This is a proposal for an admin tool - say, kafka-brokers.sh to provide 
> broker related useful information. See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
>  for details.
> The kafka-brokers.sh command mimics the kafka-topic.sh command, but provides 
> details by broker rather than by topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-3663) Proposal for a kafka broker command - kafka-brokers.sh

2016-07-03 Thread Jayesh Thakrar (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh Thakrar updated KAFKA-3663:
--
Status: Open  (was: Patch Available)

Patch not created yet, hence not avaiable.

> Proposal for a kafka broker command - kafka-brokers.sh
> --
>
> Key: KAFKA-3663
> URL: https://issues.apache.org/jira/browse/KAFKA-3663
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>        Reporter: Jayesh Thakrar
>
> This is a proposal for an admin tool - say, kafka-brokers.sh to provide 
> broker related useful information. See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
>  for details.
> The kafka-brokers.sh command mimics the kafka-topic.sh command, but provides 
> details by broker rather than by topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (KAFKA-3663) Proposal for a kafka broker command - kafka-brokers.sh

2016-07-03 Thread Jayesh Thakrar (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360809#comment-15360809
 ] 

Jayesh Thakrar edited comment on KAFKA-3663 at 7/4/16 4:04 AM:
---

Patch not created yet, hence not available.


was (Author: jthakrar):
Patch not created yet, hence not avaiable.

> Proposal for a kafka broker command - kafka-brokers.sh
> --
>
> Key: KAFKA-3663
> URL: https://issues.apache.org/jira/browse/KAFKA-3663
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>        Reporter: Jayesh Thakrar
>
> This is a proposal for an admin tool - say, kafka-brokers.sh to provide 
> broker related useful information. See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
>  for details.
> The kafka-brokers.sh command mimics the kafka-topic.sh command, but provides 
> details by broker rather than by topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-3663) Proposal for a kafka broker command - kafka-brokers.sh

2016-06-30 Thread Jayesh Thakrar (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh Thakrar updated KAFKA-3663:
--
Status: Patch Available  (was: Open)

Pull request - https://github.com/apache/kafka/pull/1539

> Proposal for a kafka broker command - kafka-brokers.sh
> --
>
> Key: KAFKA-3663
> URL: https://issues.apache.org/jira/browse/KAFKA-3663
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>        Reporter: Jayesh Thakrar
>
> This is a proposal for an admin tool - say, kafka-brokers.sh to provide 
> broker related useful information. See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
>  for details.
> The kafka-brokers.sh command mimics the kafka-topic.sh command, but provides 
> details by broker rather than by topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Kafka:3663 : Help with Jenkin Check Failures in Pull Request on Github

2016-06-29 Thread Jayesh Thakrar
Hi,
I submitted a pull request on Github - https://github.com/apache/kafka/pull/1539
Apparently the pull request shows a Jenkins failure - but without any details.
Can anyone help me identify the issue (if any) or point in the right direction?
Thanks,Jayesh


[jira] [Commented] (KAFKA-3663) Proposal for a kafka broker command - kafka-brokers.sh

2016-06-22 Thread Jayesh Thakrar (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345240#comment-15345240
 ] 

Jayesh Thakrar commented on KAFKA-3663:
---

I think I need some pointers/help here.
Apparently the pull request shows a Jenkins error, although I verified that the 
build works.
Here's the link to the pull request - https://github.com/apache/kafka/pull/1539

> Proposal for a kafka broker command - kafka-brokers.sh
> --
>
> Key: KAFKA-3663
> URL: https://issues.apache.org/jira/browse/KAFKA-3663
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>        Reporter: Jayesh Thakrar
>
> This is a proposal for an admin tool - say, kafka-brokers.sh to provide 
> broker related useful information. See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
>  for details.
> The kafka-brokers.sh command mimics the kafka-topic.sh command, but provides 
> details by broker rather than by topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-3663) Proposal for a kafka broker command - kafka-brokers.sh

2016-06-19 Thread Jayesh Thakrar (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh Thakrar updated KAFKA-3663:
--
Description: 
This is a proposal for an admin tool - say, kafka-brokers.sh to provide broker 
related useful information. See 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
 for details.

The kafka-brokers.sh command mimics the kafka-topic.sh command, but provides 
details by broker rather than by topic.


  was:
This is a proposal for an admin tool - say, kafka-brokers.sh to provide broker 
related useful information. Note that I could not see an option to create a 
child page at 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals.

So here are the details for the proposals.

*Motivation*
Some of the key succcess factor for Kafka's success are its performant 
architecture and operational simplicity. This is further complemented with a 
set of commandline tools and utilities for managing topics as well as 
testing/stress-testing. However currently Kafka lacks commands/tools to get a 
cluster and broker overview. Although it should be mentioned that Kafka does 
expose cluster information via API and broker metrics via JMX.

*Proposed Change*
This KIP is for a command, say kafka-brokers.sh that provides useful cluster 
and broker information. 

The command will essentially provide the following pieces of information:

* Cluster Overview Information
** Controller Broker Id (and version/epoch information)
** Broker Count
** Total Topic Count
** Total Partition Count

* Broker Information
** Broker Id
** Rack Id
** Hostname
** Endpoints (protocol, port)
** JMX port
** Topic count
** All partition count
** Leader partition count
** Under-replicated partition count
** Topic partitions (Name, *partition-) - 
An asterisk would indicate that the broker is the leader for that partition.
A hyphen/negative sign would indicates that the partition is not in-sync.


As you can see, the above information provides a view of the cluster and 
brokers that complements kafka-topics.sh.

This command can be further evolved to do more things like:
- Drain one or more brokers for decommissioning. This feature would allow 
distributing off all partitions of a list of brokers to other brokers in the 
cluster (need to be cognizant of rack configuration). 
- Onboard one or more new brokers. This feature would do the reverse of 
removing brokers and allow controlled distribution of partition to the new 
brokers.
- Obtain JMX metrics from a broker (see KIP for its usage).
- See KAFKA-3649 for other features



> Proposal for a kafka broker command - kafka-brokers.sh
> --
>
> Key: KAFKA-3663
> URL: https://issues.apache.org/jira/browse/KAFKA-3663
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>        Reporter: Jayesh Thakrar
>
> This is a proposal for an admin tool - say, kafka-brokers.sh to provide 
> broker related useful information. See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
>  for details.
> The kafka-brokers.sh command mimics the kafka-topic.sh command, but provides 
> details by broker rather than by topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

2016-05-09 Thread Jayesh Thakrar
Hi All,



This is to start off a discussion on the above KIP at 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command
The proposal is to fill the void of a command line tool/utility that can 
provide information on the cluster and brokers in a Kafka cluster.
Thank you,Jayesh Thakrar

  

[DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

2016-05-09 Thread Jayesh Thakrar
Hi All,
This is to start off a discussion on the above KIP at 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-59+-+Proposal+for+a+kafka+broker+command+-+kafka-brokers.sh
The proposal is to fill the void of a command line tool/utility that can 
provide information on the cluster and brokers in a Kafka cluster.
Thank you,Jayesh Thakrar

Re: How to submit a KIP?

2016-05-07 Thread Jayesh Thakrar
Thank you Ismael.My Apache id is jthakrar.
Also, if you can let me know what the next step would be after creating the KIP.
Thanks,Jayesh

  From: Ismael Juma <ism...@juma.me.uk>
 To: dev@kafka.apache.org; Jayesh Thakrar <j_thak...@yahoo.com> 
 Sent: Saturday, May 7, 2016 2:57 AM
 Subject: Re: How to submit a KIP?
   
Hi Jayesh,
You need to create an Apache wiki account (if you haven't already) and then 
share your wiki id so that one of us can give you the required permissions.
Thanks,Ismael
On Sat, May 7, 2016 at 3:44 AM, Jayesh Thakrar <j_thak...@yahoo.com.invalid> 
wrote:

Hi,
I have created a Jira Issue - https://issues.apache.org/jira/browse/KAFKA-3663 
and was wondering what is the next step?Should I create a KIP? If so, how do I 
do that, as I do not have permissions to do that.
Thanks,Jayesh Thakrar




  

How to submit a KIP?

2016-05-06 Thread Jayesh Thakrar
Hi,
I have created a Jira Issue - https://issues.apache.org/jira/browse/KAFKA-3663 
and was wondering what is the next step?Should I create a KIP? If so, how do I 
do that, as I do not have permissions to do that.
Thanks,Jayesh Thakrar


[jira] [Created] (KAFKA-3663) Proposal for a kafka broker command - kafka-brokers.sh

2016-05-05 Thread Jayesh Thakrar (JIRA)
Jayesh Thakrar created KAFKA-3663:
-

 Summary: Proposal for a kafka broker command - kafka-brokers.sh
 Key: KAFKA-3663
 URL: https://issues.apache.org/jira/browse/KAFKA-3663
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Reporter: Jayesh Thakrar


This is a proposal for an admin tool - say, kafka-brokers.sh to provide broker 
related useful information. Note that I could not see an option to create a 
child page at 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals.

So here are the details for the proposals.

*Motivation*
Some of the key succcess factor for Kafka's success are its performant 
architecture and operational simplicity. This is further complemented with a 
set of commandline tools and utilities for managing topics as well as 
testing/stress-testing. However currently Kafka lacks commands/tools to get a 
cluster and broker overview. Although it should be mentioned that Kafka does 
expose cluster information via API and broker metrics via JMX.

*Proposed Change*
This KIP is for a command, say kafka-brokers.sh that provides useful cluster 
and broker information. 

The command will essentially provide the following pieces of information:

* Cluster Overview Information
** Controller Broker Id (and version/epoch information)
** Broker Count
** Total Topic Count
** Total Partition Count

* Broker Information
** Broker Id
** Rack Id
** Hostname
** Endpoints (protocol, port)
** JMX port
** Topic count
** All partition count
** Leader partition count
** Under-replicated partition count
** Topic partitions (Name, *partition-) - 
An asterisk would indicate that the broker is the leader for that partition.
A hyphen/negative sign would indicates that the partition is not in-sync.


As you can see, the above information provides a view of the cluster and 
brokers that complements kafka-topics.sh.

This command can be further evolved to do more things like:
- Drain one or more brokers for decommissioning. This feature would allow 
distributing off all partitions of a list of brokers to other brokers in the 
cluster (need to be cognizant of rack configuration). 
- Onboard one or more new brokers. This feature would do the reverse of 
removing brokers and allow controlled distribution of partition to the new 
brokers.
- Obtain JMX metrics from a broker (see KIP for its usage).
- See KAFKA-3649 for other features




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: KIP Call Notes

2015-06-10 Thread Jayesh Thakrar
Our company (Conversant Media) became a Kafka user pretty recently (about 3 
months ago) and I have been following the user and dev group for the past 
month, so could have missed the discussions on KIP-4.
In looking up the current command line capabilities in Kafka 0.8.2.1 (not the 
latest release) it seems we are missing a couple of pieces.
Ideally, I am looking at getting the following kind of information
Describe a Kafka cluster (example) -kafka cluster metadata = {
  brokers : [ { id : 0, attributes from broker Json schema}, { id : 1, 
attributes from broker Json schema}, ...]  topics : [ { name : topic1, 
attributes from topic and parition and ISR and leader information }, { name : 
topic2, attributes from topic and parition and ISR and leader information } ] 
}

Topic metrics = atleast some kind of message high water mark from the leaders 
for each topic partition
Describe consumers (example) -Consumer metadata = {     [ group : 
group_id1, client : client_id1, [ list of subscribed topics ]  ], ..}
Having such information will make it easy to build monitoring tools and also 
aid troubleshooting and sizing/configuration.
Is discussing these things as part of KIP the right process or is there some 
other forum?
Thanks,Jayesh Thakrar


  From: Gwen Shapira gshap...@cloudera.com
 To: dev@kafka.apache.org dev@kafka.apache.org 
 Sent: Tuesday, June 9, 2015 1:19 PM
 Subject: KIP Call Notes
   
Shortest KIP discussion ever :)

Here are my notes. Feel free to correct or expand:

SSL: Harsh is rebasing after Kafka-1928, and resolving an issue with
support for specific API

KIP-4  - ISR in Metadata request: We prefer to preserve this information.

KIP-25 - System Tests: Geoffrey posted an update with discussion
summary, there were no objections to the proposal. We will wait a day
before starting a vote.

New consumer - Separate the API changes for metadata to a new JIRA.
Since the new consumer is still work in progress, there's no need for
KIP for every modification.


  

Newbie question on graddle startup error

2015-06-04 Thread Jayesh Thakrar
Hi,
I have been using Kafka as DevOps user for a few months now and am now looking 
at building/testing Kafka from source code.My gradle build is failing (source 
from GitHub) with the error show below.This is a windows laptop with 4 GB 
memory and about 3 GB free RAM.I have also tried increasing the heap size 
(-Xmx) to 1500m, but that did not help too.
I am sure this is something very basic, but can't seem to be able to figure 
out.Thanks,Jayesh

C:\Users\jthakrar\Jayesh\Downloads\kafka\kafkagradle --info 
--debug14:00:39.314 [LIFECYCLE] 
[org.gradle.launcher.daemon.client.SingleUseDaemonClient] To honour the JVM 
settings for this build a new JVM will be forked. Please consider using the 
daemon: http://gradle.org/docs/2.2.1/userguide/gradle_daemon.html.14:00:39.318 
[INFO] [org.gradle.launcher.daemon.client.DefaultDaemonConnector] Starting 
Gradle daemon14:00:39.336 [DEBUG] 
[org.gradle.launcher.daemon.client.DefaultDaemonStarter] Using daemon opts: 
[-XX:MaxPermSize=512m, -Xmx1024m, -Dfile.encoding=windows-1252, 
-Duser.country=US, -Duser.language=en, -Duser.variant]14:00:39.337 [INFO] 
[org.gradle.launcher.daemon.client.DefaultDaemonStarter] Starting daemon 
process: workingDir = C:\Users\jthakrar\.gradle\daemon\2.2.1, daemonArgs: 
[C:\Program Files (x86)\Java\jdk1.7.0_07\bin\java.exe, -XX:MaxPermSize=512m, 
-Xmx1024m, -Dfile.encoding=windows-1252, -Duser.country=US, -Duser.language=en, 
-Duser.variant, -cp, 
C:\Users\jthakrar\Jayesh\Downloads\Graddle\gradle-2.2.1\lib\gradle-launcher-2.2.1.jar,
 org.gradle.launcher.daemon.bootstrap.GradleDaemon, 2.2.1, 
C:\Users\jthakrar\.gradle\daemon, 12, b71a3ff7-2bc6-442d-a470-a5be0eec3b37, 
-XX:MaxPermSize=512m, -Xmx1024m, -Dfile.encoding=windows-1252, 
-Duser.country=US, -Duser.language=en, -Duser.variant]14:00:39.357 [INFO] 
[org.gradle.process.internal.DefaultExecHandle] Starting process 'Gradle build 
daemon'. Working directory: C:\Users\jthakrar\.gradle\daemon\2.2.1 
Command:C:\Program Files (x86)\Java\jdk1.7.0_07\bin\java.exe 
-XX:MaxPermSize=512m -Xmx1024m -Dfile.encoding=windows-1252 -Duser.country=US 
-Duser.language=en -Duser.variant -cp 
C:\Users\jthakrar\Jayesh\Downloads\Graddle\gradle-2.2.1\lib\gradle-launcher-2.2.1.jar
 org.gradle.launcher.daemon.bootstrap.GradleDaemon 2.2.1 
C:\Users\jthakrar\.gradle\daemon 12 b71a3ff7-2bc6-442d-a470-a5be0eec3b37 
-XX:MaxPermSize=512m -Xmx1024m -Dfile.encoding=windows-1252 -Duser.country=US 
-Duser.language=en -Duser.variant14:00:39.360 [DEBUG] 
[org.gradle.process.internal.DefaultExecHandle] Environment for process 'Gradle 
build daemon': {CMD_LINE_ARGS=--info --debug, USERPROFILE=C:\Users\jthakrar, 
DIRNAME=C:\Users\jthakrar\Jayesh\Downloads\Graddle\gradle-2.2.1\bin\, 
JAVA_HOME=C:\Program Files (x86)\Java\jdk1.7.0_07, SystemDrive=C:, 
M2=C:\Program Files\Apache Software Foundation\apache-maven-3.0.5\bin, 
Path=C:\Ruby22-x64\bin;C:\Program Files\Common Files\Microsoft Shared\Windows 
Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows 
Live;c:\Rtools\bin;c:\Rtools\gcc-4.6.3\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program
 Files\Apache Software Foundation\apache-maven-3.0.5\bin;C:\Python27;C:\Program 
Files (x86)\QuickTime\QTSystem\;C:\Program Files (x86)\Git\cmd;C:\Program 
Files\TortoiseGit\bin;C:\Users\jthakrar\Jayesh\Downloads\Subversion\Apache-Subversion-1.8.0\bin;C:\Program
 Files 
(x86)\Java\jdk1.7.0_07\bin;C:\Anaconda;C:\Anaconda\Scripts;C:\Users\jthakrar\Jayesh\Downloads\Protobuf\protoc-2.5.0-win32;C:\Program
 Files (x86)\Windows Live\Shared;C:\Program Files (x86)\sbt;C:\Program Files 
(x86)\sbt\\bin;C:\Users\jthakrar\Jayesh\Downloads\MongoDB\mongodb-win32-x86_64-2008plus-2.6.5\mongodb-win32-x86_64-2008plus-2.6.5\bin;C:\Users\jthakrar\Jayesh\Downloads\Storm\apache-storm-0.9.3-src.tar\apache-storm-0.9.3\bin;C:\Program
 Files 
(x86)\scala\bin;C:\Users\jthakrar\Jayesh\Downloads\Graddle\gradle-2.2.1\bin;C:\Users\jthakrar\Jayesh\Downloads\Graddle\gradle-2.2.1\bin,
 _SKIP=2, PROCESSOR_REVISION=3a09, USERDOMAIN=CORP, 
ALLUSERSPROFILE=C:\ProgramData, SESSIONNAME=Console, M2_OPTS=-Xms256m -Xmx512m, 
TMP=C:\Users\jthakrar\AppData\Local\Temp, M2_HOME=C:\Program Files\Apache 
Software Foundation\apache-maven-3.0.5, LOGONSERVER=\\CHI-DC102, 
CommonProgramFiles=C:\Program Files (x86)\Common Files, PROMPT=$P$G, 
PROCESSOR_LEVEL=6, LOCALAPPDATA=C:\Users\jthakrar\AppData\Local, 
COMPUTERNAME=CHI-JTHAKRAR, SystemRoot=C:\WINDOWS, USERNAME=jthakrar, 
APP_BASE_NAME=gradle, APPDATA=C:\Users\jthakrar\AppData\Roaming, 
ProgramData=C:\ProgramData, 
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC, 
USERDNSDOMAIN=CORP.VALUECLICK.COM, JAVA_EXE=C:\Program Files 
(x86)\Java\jdk1.7.0_07/bin/java.exe, 
windows_tracing_logfile=C:\BVTBin\Tests\installpackage\csilogfile.log, 
ProgramFiles(x86)=C:\Program Files (x86), 
APP_HOME=C:\Users\jthakrar\Jayesh\Downloads\Graddle\gradle-2.2.1\bin\.., 
TEMP=C:\Users\jthakrar\AppData\Local\Temp,