Re: Review Request 24676: Fix KAFKA-1583

2014-10-17 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24676/#review57147 --- Thanks for the patch. A few more comments.

[jira] [Updated] (KAFKA-1583) Kafka API Refactoring

2014-10-17 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1583: - Attachment: KAFKA-1583_2014-10-17_09:56:33.patch Kafka API Refactoring -

[jira] [Created] (KAFKA-1711) WARN Property topic is not valid when running console producer

2014-10-17 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-1711: -- Summary: WARN Property topic is not valid when running console producer Key: KAFKA-1711 URL: https://issues.apache.org/jira/browse/KAFKA-1711 Project: Kafka Issue

[jira] [Commented] (KAFKA-1583) Kafka API Refactoring

2014-10-17 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175239#comment-14175239 ] Guozhang Wang commented on KAFKA-1583: -- Updated reviewboard

RE: Security JIRAS

2014-10-17 Thread Arvind Mani
I'm looking at Kafka Brokers authentication with ZooKeeper since this looks independent of other tasks. [AM] 1) Is authentication required only between kafka broker and zookeeper? Can we assume world read so that consumers don't have to be authenticated (I believe in any case kafka is

[jira] [Updated] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

2014-10-17 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1493: --- Resolution: Fixed Status: Resolved (was: Patch Available) James, Thanks a lot for the patch. +1 and

Jenkins build is back to normal : Kafka-trunk #305

2014-10-17 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/305/changes

Re: Security JIRAS

2014-10-17 Thread Todd Palino
For the moment, consumers still need to write under the /consumers tree. Even if they are committing offsets to Kafka instead of ZK, they will need to write owner information there when they are balancing. Eventually, you are correct, this is going away with the new consumer. -Todd On Fri, Oct

Re: Security JIRAS

2014-10-17 Thread Gwen Shapira
Yes, I think we can focus on Broker to Zookeeper communication only. At least for initial stage. Gwen On Fri, Oct 17, 2014 at 2:10 PM, Todd Palino tpal...@gmail.com wrote: For the moment, consumers still need to write under the /consumers tree. Even if they are committing offsets to Kafka

Re: Review Request 24676: Rebase KAFKA-1583

2014-10-17 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24676/#review57178 --- Thanks for the patch. +1 after addressing a couple of more minor

[jira] [Updated] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2014-10-17 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1642: - Assignee: Ewen Cheslack-Postava (was: Jun Rao) Status: Patch Available

[jira] [Commented] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2014-10-17 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175360#comment-14175360 ] Ewen Cheslack-Postava commented on KAFKA-1642: -- Created reviewboard

Review Request 26885: Patch for KAFKA-1642

2014-10-17 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26885/ --- Review request for kafka. Bugs: KAFKA-1642

[jira] [Updated] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2014-10-17 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1642: - Attachment: KAFKA-1642.patch [Java New Producer Kafka Trunk] CPU Usage Spike to

[jira] [Commented] (KAFKA-1485) Upgrade to Zookeeper 3.4.6 and create shim for ZKCLI so system tests can run

2014-10-17 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175365#comment-14175365 ] Jun Rao commented on KAFKA-1485: Gwen, Joe, It seems that ZK 3.4.6 added a runtime

[jira] [Created] (KAFKA-1712) Excessive storage usage on newly added node

2014-10-17 Thread Oleg Golovin (JIRA)
Oleg Golovin created KAFKA-1712: --- Summary: Excessive storage usage on newly added node Key: KAFKA-1712 URL: https://issues.apache.org/jira/browse/KAFKA-1712 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-1671) uploaded archives are missing for Scala version 2.11

2014-10-17 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1671: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. +1. Committed to

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175430#comment-14175430 ] Ewen Cheslack-Postava commented on KAFKA-1710: -- bq. The dead lock will occur

Build failed in Jenkins: Kafka-trunk #306

2014-10-17 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/306/changes Changes: [junrao] kafka-1671; uploaded archives are missing for Scala version 2.11; patched by Ivan Lyutov; reviewed by Jun Rao -- [...truncated 1011 lines...] kafka.producer.ProducerTest

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

2014-10-17 Thread BalajiSeshadri (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175500#comment-14175500 ] BalajiSeshadri commented on KAFKA-1476: --- [~nehanarkhede] I tried to implement the

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

2014-10-17 Thread BalajiSeshadri (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BalajiSeshadri updated KAFKA-1476: -- Attachment: ConsumerCommand.scala Get a list of consumer groups -

[jira] [Resolved] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-1710. -- Resolution: Invalid Assignee: Ewen Cheslack-Postava [New Java Producer

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

2014-10-17 Thread BalajiSeshadri (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175503#comment-14175503 ] BalajiSeshadri commented on KAFKA-1476: --- [~nehanarkhede] Please find the attached

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Bhavesh Mistry (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175548#comment-14175548 ] Bhavesh Mistry commented on KAFKA-1710: --- [~ewencp], Thank you for entertaining this

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175551#comment-14175551 ] Jay Kreps commented on KAFKA-1710: -- [~Bmis13] What is the performance you see? What do

[jira] [Comment Edited] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Bhavesh Mistry (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175548#comment-14175548 ] Bhavesh Mistry edited comment on KAFKA-1710 at 10/17/14 9:19 PM:

blog with some out of the box pains

2014-10-17 Thread Jay Kreps
This guy documented a few struggles getting going with Kafka. Not sure if there is anything we can do to make it better? http://ispyker.blogspot.com/2014/10/kafka-part-1.html 1. Would be great to figure out the apache/gradle thing. 2. The problem of having Kafka advertise localhost on AWS is

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Bhavesh Mistry (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175574#comment-14175574 ] Bhavesh Mistry commented on KAFKA-1710: --- [~jkreps], Only request is to make New

[jira] [Comment Edited] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Bhavesh Mistry (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175574#comment-14175574 ] Bhavesh Mistry edited comment on KAFKA-1710 at 10/17/14 9:32 PM:

[jira] [Commented] (KAFKA-1710) [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-17 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175586#comment-14175586 ] Jay Kreps commented on KAFKA-1710: -- Well but of course you can't have multiple threads

[jira] [Updated] (KAFKA-1108) when controlled shutdown attempt fails, the reason is not always logged

2014-10-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1108: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. Pushed

[jira] [Updated] (KAFKA-1108) when controlled shutdown attempt fails, the reason is not always logged

2014-10-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1108: - Fix Version/s: (was: 0.9.0) 0.8.2 when controlled shutdown attempt fails,

Re: blog with some out of the box pains

2014-10-17 Thread Gwen Shapira
In #2, do you refer to advertising the internal hostname instead of the external one? In this case, will it be enough to use getCanonicalHostName (which uses a name service)? Note that I think the problem the blog reported (wrong name advertised) is somewhat orthogonal to the question of which

Jenkins build is back to normal : Kafka-trunk #307

2014-10-17 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/307/changes

[jira] [Created] (KAFKA-1713) Partition files not created

2014-10-17 Thread Pradeep (JIRA)
Pradeep created KAFKA-1713: -- Summary: Partition files not created Key: KAFKA-1713 URL: https://issues.apache.org/jira/browse/KAFKA-1713 Project: Kafka Issue Type: Bug Components: core

[jira] [Resolved] (KAFKA-1713) Partition files not created

2014-10-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-1713. -- Resolution: Won't Fix Please can you redirect such questions to the mailing list? Regarding

Re: blog with some out of the box pains

2014-10-17 Thread Jay Kreps
Hmm, yes, actually I don't think I actually understand the issue. Basically as I understand it we do InetAddress.getLocalHost.getHostAddress which on AWS picks the wrong hostname/ip and then the producer can't connect. People eventually find this FAQ, but I was hoping there was a more automatic

Kafka Command Line Shell

2014-10-17 Thread Joe Stein
Hi, I have been thinking about the ease of use for operations with Kafka. We have lots of tools doing a lot of different things and they are all kind of in different places. So, what I was thinking is to have a single interface for our tooling https://issues.apache.org/jira/browse/KAFKA-1694

Re: Kafka Command Line Shell

2014-10-17 Thread Steve Morin
Joe I think this is great! On Fri, Oct 17, 2014 at 5:03 PM, Joe Stein joe.st...@stealth.ly wrote: Hi, I have been thinking about the ease of use for operations with Kafka. We have lots of tools doing a lot of different things and they are all kind of in different places. So, what I was

Re: blog with some out of the box pains

2014-10-17 Thread Gwen Shapira
Basically, the issue (or at least one of very many possible network issues...) is that the server has localhost hardcoded as its canonical name in /etc/hosts: [root@Billc-cent70x64 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 Billc-cent70x64

Re: Kafka Command Line Shell

2014-10-17 Thread Todd Palino
We've been talking about this a little internally as well. What about the idea of presenting all the admin functions through a web API interface (restful or not) complete with authentication? That would make it much easier for creating structure around Kafka without having to layer commands on

[jira] [Commented] (KAFKA-1583) Kafka API Refactoring

2014-10-17 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175763#comment-14175763 ] Joel Koshy commented on KAFKA-1583: --- Reviewing is taking a bit longer than expected -

Re: Kafka Command Line Shell

2014-10-17 Thread Joel Koshy
+1 It would definitely be useful to have a CLI. We had a cursory discussion on this in the past [1] but it would be useful to have a full proposal describing everything the CLI should provide. [1] http://grokbase.com/t/kafka/dev/1435tr3pfc/command-line-tools On Fri, Oct 17, 2014 at 05:12:16PM

Re: Kafka Command Line Shell

2014-10-17 Thread Todd Palino
Absolutely. My suggestion of an HTTP interface was in addition to a CLI. I think the CLI should can the HTTP interface underneath to keep it simple. -Todd On Oct 17, 2014, at 6:24 PM, Joel Koshy jjkosh...@gmail.com wrote: +1 It would definitely be useful to have a CLI. We had a cursory

Re: Kafka Command Line Shell

2014-10-17 Thread gshapira
+1 to http interface and cli. Http layer will make it easier to integrate with gui like Hue. Gwen — Sent from Mailbox On Fri, Oct 17, 2014 at 10:14 PM, Todd Palino tpal...@gmail.com wrote: Absolutely. My suggestion of an HTTP interface was in addition to a CLI. I think the CLI should can

Re: blog with some out of the box pains

2014-10-17 Thread Ewen Cheslack-Postava
The first issue he runs into is one I also find frustrating -- with cloud providers pushing SSDs, you have to use a pretty large instance type to get a reasonable test setup. I'm not sure if he couldn't launch an older type like m1.large (I think some newer AWS accounts aren't able to) or if he