Re: Review Request 26663: Patch for KAFKA-979

2014-10-14 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26663/#review56494 --- core/src/main/scala/kafka/log/Log.scala

Re: [DISCUSSION] Message Metadata

2014-10-14 Thread Joe Stein
I think we could add schemaId(binary) to the MessageAndMetaData With the schemaId you can implement different downstream software pattern on the messages reliably. I wrote up more thoughts on this use https://cwiki.apache.org/confluence/display/KAFKA/Schema+based+topics it should strive to

[jira] [Updated] (KAFKA-1691) new java consumer needs ssl support as a client

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1691: - Component/s: security new java consumer needs ssl support as a client

[jira] [Updated] (KAFKA-1682) Security for Kafka

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1682: - Component/s: security Security for Kafka -- Key: KAFKA-1682

[jira] [Updated] (KAFKA-1685) Implement TLS/SSL tests

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1685: - Component/s: security Implement TLS/SSL tests --- Key: KAFKA-1685

[jira] [Updated] (KAFKA-1684) Implement TLS/SSL authentication

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1684: - Component/s: security Implement TLS/SSL authentication

[jira] [Updated] (KAFKA-1686) Implement SASL/Kerberos

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1686: - Component/s: security Implement SASL/Kerberos --- Key: KAFKA-1686

[jira] [Updated] (KAFKA-1683) Implement a session concept in the socket server

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1683: - Component/s: security Implement a session concept in the socket server

[jira] [Updated] (KAFKA-1695) Authenticate connection to Zookeeper

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1695: - Component/s: security Authenticate connection to Zookeeper

[jira] [Updated] (KAFKA-1477) add authentication layer and initial JKS x509 implementation for brokers, producers and consumer for network communication

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1477: - Component/s: security add authentication layer and initial JKS x509 implementation for brokers,

[jira] [Updated] (KAFKA-1696) Kafka should be able to generate Hadoop delegation tokens

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1696: - Component/s: security Kafka should be able to generate Hadoop delegation tokens

[jira] [Updated] (KAFKA-1688) Add authorization interface and naive implementation

2014-10-14 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1688: - Component/s: security Add authorization interface and naive implementation

request for edit access to Kafka wiki

2014-10-14 Thread Glen Mazza
Hi, I'm Glen Mazza from the Apache Roller and CXF teams. Would I be able to have write access to the Kafka wiki? (I'm mazzag with Confluence, I already have an ICLA filed with Apache.) I'm starting to go through the program and its tutorials and would like to make updates if I see things

Re: request for edit access to Kafka wiki

2014-10-14 Thread Joe Stein
Glen, I just added them, you should be good to go! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

2014-10-14 Thread James Oliver (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171082#comment-14171082 ] James Oliver commented on KAFKA-1493: - Sorry to not be more clear - I fixed a few

Re: Review Request 26663: Patch for KAFKA-979

2014-10-14 Thread Neha Narkhede
On Oct. 14, 2014, 6:27 a.m., Joel Koshy wrote: core/src/main/scala/kafka/log/Log.scala, line 515 https://reviews.apache.org/r/26663/diff/1/?file=719793#file719793line515 Although this should work I'm wondering if it would be better to recompute a random jitter for each new rolled

Re: Review Request 26663: Patch for KAFKA-979

2014-10-14 Thread Joel Koshy
On Oct. 14, 2014, 6:27 a.m., Joel Koshy wrote: core/src/main/scala/kafka/log/Log.scala, line 515 https://reviews.apache.org/r/26663/diff/1/?file=719793#file719793line515 Although this should work I'm wondering if it would be better to recompute a random jitter for each new rolled

[jira] [Assigned] (KAFKA-566) Add last modified time to the TopicMetadataRequest

2014-10-14 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani reassigned KAFKA-566: Assignee: Sriharsha Chintalapani Add last modified time to the

[jira] [Assigned] (KAFKA-560) Garbage Collect obsolete topics

2014-10-14 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani reassigned KAFKA-560: Assignee: Sriharsha Chintalapani Garbage Collect obsolete topics

[jira] [Commented] (KAFKA-560) Garbage Collect obsolete topics

2014-10-14 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171169#comment-14171169 ] Sriharsha Chintalapani commented on KAFKA-560: -- [~jkreps] [~nehanarkhede] If

Re: [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-14 Thread Jay Kreps
Hey Bhavesh, This sounds like a problem. Just to confirm this is after the fix for KAFKA-1673? https://issues.apache.org/jira/browse/KAFKA-1673 It sounds like you have a reproducible test case? -Jay On Mon, Oct 13, 2014 at 10:54 AM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi

Re: [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-14 Thread Bhavesh Mistry
HI Jay, Yes, it is reproducible quite easily. The problem is synchronized in RecordAccumulator. You can easy produce it. I have attached the Java code in my original email. Due to Application threads enqueue message into single partition is causing thrad contention and application thread may

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-14 Thread Vladimir Tretyakov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171371#comment-14171371 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, Jun, Thx that you found a

[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-14 Thread Vladimir Tretyakov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch

[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-14 Thread Vladimir Tretyakov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171371#comment-14171371 ] Vladimir Tretyakov edited comment on KAFKA-1481 at 10/14/14 7:09 PM:

Re: Review Request 26663: Patch for KAFKA-979

2014-10-14 Thread Neha Narkhede
On Oct. 14, 2014, 6:27 a.m., Joel Koshy wrote: core/src/main/scala/kafka/log/Log.scala, line 515 https://reviews.apache.org/r/26663/diff/1/?file=719793#file719793line515 Although this should work I'm wondering if it would be better to recompute a random jitter for each new rolled

[jira] [Commented] (KAFKA-566) Add last modified time to the TopicMetadataRequest

2014-10-14 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171531#comment-14171531 ] Neha Narkhede commented on KAFKA-566: - [~sriharsha] Thanks for taking this on. Add

[jira] [Updated] (KAFKA-566) Add last modified time to the TopicMetadataRequest

2014-10-14 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-566: Reviewer: Neha Narkhede Add last modified time to the TopicMetadataRequest

[jira] [Updated] (KAFKA-560) Garbage Collect obsolete topics

2014-10-14 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-560: Reviewer: Neha Narkhede Garbage Collect obsolete topics ---

Re: Review Request 26663: Patch for KAFKA-979

2014-10-14 Thread Neha Narkhede
On Oct. 14, 2014, 6:27 a.m., Joel Koshy wrote: core/src/main/scala/kafka/log/Log.scala, line 515 https://reviews.apache.org/r/26663/diff/1/?file=719793#file719793line515 Although this should work I'm wondering if it would be better to recompute a random jitter for each new rolled

Review Request 26710: Patch for KAFKA-1637

2014-10-14 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26710/ --- Review request for kafka. Bugs: KAFKA-1637

[jira] [Updated] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

2014-10-14 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1637: - Assignee: Ewen Cheslack-Postava Status: Patch Available (was: Open)

[jira] [Updated] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

2014-10-14 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1637: - Attachment: KAFKA-1637.patch SimpleConsumer.fetchOffset returns wrong error code

[jira] [Commented] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

2014-10-14 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171610#comment-14171610 ] Ewen Cheslack-Postava commented on KAFKA-1637: -- Created reviewboard

[jira] [Commented] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

2014-10-14 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171621#comment-14171621 ] Ewen Cheslack-Postava commented on KAFKA-1637: -- The error code is for

Re: Review Request 26663: Patch for KAFKA-979

2014-10-14 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26663/ --- (Updated Oct. 14, 2014, 10:33 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-979) Add jitter for time based rolling

2014-10-14 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171648#comment-14171648 ] Ewen Cheslack-Postava commented on KAFKA-979: - Updated reviewboard

[jira] [Updated] (KAFKA-979) Add jitter for time based rolling

2014-10-14 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-979: Attachment: KAFKA-979_2014-10-14_15:33:31.patch Add jitter for time based rolling

Re: [New Java Producer Potential Deadlock] Producer Deadlock when all messages is being sent to single partition

2014-10-14 Thread Jun Rao
Bhavesh, It seems that all those threads are blocked on waiting for the lock on the dq for that partition. There got to be another thread holding the dq lock at that point. Could you create a jira and attach the full thread dump there? Also, could you attach the yourkit result that shows the

[jira] [Created] (KAFKA-1705) Add MR layer to Kafka

2014-10-14 Thread Gwen Shapira (JIRA)
Gwen Shapira created KAFKA-1705: --- Summary: Add MR layer to Kafka Key: KAFKA-1705 URL: https://issues.apache.org/jira/browse/KAFKA-1705 Project: Kafka Issue Type: Improvement

[jira] [Commented] (KAFKA-1705) Add MR layer to Kafka

2014-10-14 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171720#comment-14171720 ] Gwen Shapira commented on KAFKA-1705: - Looking for community comments on this. Do

[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

2014-10-14 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171732#comment-14171732 ] Jun Rao commented on KAFKA-1493: James, Thanks, got it now. Not sure how long it will

[jira] [Commented] (KAFKA-1493) Use a well-documented LZ4 compression format and remove redundant LZ4HC option

2014-10-14 Thread James Oliver (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171748#comment-14171748 ] James Oliver commented on KAFKA-1493: - I implemented the OutputStream today. If I

Bad link on the document

2014-10-14 Thread Azuryy Yu
Hi, Usage information on the hadoop consumer can be found here https://github.com/linkedin/camus/tree/camus-kafka-0.8/. the Link is broken, who can kindly fix it, thanks

Re: Review Request 26663: Patch for KAFKA-979

2014-10-14 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26663/#review56652 --- Looks good overall - I have one comment on interpreting the jitter

[jira] [Updated] (KAFKA-979) Add jitter for time based rolling

2014-10-14 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-979: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. Pushed to

[jira] [Commented] (KAFKA-979) Add jitter for time based rolling

2014-10-14 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171962#comment-14171962 ] Joel Koshy commented on KAFKA-979: -- Just had one more comment in the RB on how to use the

Re: Review Request 26710: Patch for KAFKA-1637

2014-10-14 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26710/#review56655 --- Ship it! Ship It! - Neha Narkhede On Oct. 14, 2014, 10:04 p.m.,

[jira] [Updated] (KAFKA-1698) Validator.ensureValid() only validates default config value

2014-10-14 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1698: - Reviewer: Jun Rao Validator.ensureValid() only validates default config value

[jira] [Updated] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

2014-10-14 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1637: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. Pushed

[jira] [Commented] (KAFKA-1583) Kafka API Refactoring

2014-10-14 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171998#comment-14171998 ] Guozhang Wang commented on KAFKA-1583: -- Bump, [~junrao] could you take a look now?

Re: Review Request 26710: Patch for KAFKA-1637

2014-10-14 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26710/#review56662 --- core/src/main/scala/kafka/server/KafkaApis.scala

[jira] [Commented] (KAFKA-1637) SimpleConsumer.fetchOffset returns wrong error code when no offset exists for topic/partition/consumer group

2014-10-14 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172020#comment-14172020 ] Joel Koshy commented on KAFKA-1637: --- -1 on this patch per the comment in the RB.