[jira] [Commented] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2014-10-20 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176624#comment-14176624 ] Guozhang Wang commented on KAFKA-1716: -- This may be resolved in KAFKA-1305. Did you

Re: [DISCUSSION] Message Metadata

2014-10-20 Thread Guozhang Wang
Thanks for the detailed comments Jun! Some replies inlined. On Sun, Oct 19, 2014 at 7:42 PM, Jun Rao jun...@gmail.com wrote: Hi, Guozhang, Thanks for the writeup. A few high level comments. 1. Associating (versioned) schemas to a topic can be a good thing overall. Yes, this could add a

[jira] [Updated] (KAFKA-1712) Excessive storage usage on newly added node

2014-10-20 Thread Oleg Golovin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Golovin updated KAFKA-1712: Description: When a new node is added to cluster data starts replicating into it. The mtime of

[jira] [Updated] (KAFKA-1712) Excessive storage usage on newly added node

2014-10-20 Thread Oleg Golovin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Golovin updated KAFKA-1712: Description: When a new node is added to cluster data starts replicating into it. The mtime of

[jira] [Commented] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2014-10-20 Thread Sean Fay (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176893#comment-14176893 ] Sean Fay commented on KAFKA-1716: - Nope, this was running 0.8.1.1. I'm not very familiar

[jira] [Commented] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2014-10-20 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176904#comment-14176904 ] Sriharsha Chintalapani commented on KAFKA-1716: --- [~sfay] do you have any

[jira] [Updated] (KAFKA-1171) Gradle build for Kafka

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1171: - Fix Version/s: 0.8.1 0.8.1.1 Gradle build for Kafka --

[jira] [Closed] (KAFKA-262) Bug in the consumer rebalancing logic causes one consumer to release partitions that it does not own

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-262. --- Bug in the consumer rebalancing logic causes one consumer to release partitions that it does not own

[jira] [Closed] (KAFKA-1171) Gradle build for Kafka

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1171. Gradle build for Kafka -- Key: KAFKA-1171 URL:

[jira] [Commented] (KAFKA-1630) ConsumerFetcherThread locked in Tomcat

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176927#comment-14176927 ] Neha Narkhede commented on KAFKA-1630: -- The thread dump you attached doesn't suggest

[jira] [Updated] (KAFKA-1714) more better bootstrapping of the gradle-wrapper.jar

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1714: - Component/s: build more better bootstrapping of the gradle-wrapper.jar

[jira] [Updated] (KAFKA-1715) better advertising of the bound and working interface

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1715: - Labels: newbie (was: ) better advertising of the bound and working interface

[jira] [Updated] (KAFKA-1714) more better bootstrapping of the gradle-wrapper.jar

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1714: - Affects Version/s: 0.8.2 more better bootstrapping of the gradle-wrapper.jar

[jira] [Commented] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176944#comment-14176944 ] Neha Narkhede commented on KAFKA-1716: -- [~sfay] Could you please attach the entire

[jira] [Created] (KAFKA-1717) remove netty dependency through ZK 3.4.x

2014-10-20 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-1717: -- Summary: remove netty dependency through ZK 3.4.x Key: KAFKA-1717 URL: https://issues.apache.org/jira/browse/KAFKA-1717 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-1717) remove netty dependency through ZK 3.4.x

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1717: --- Status: Patch Available (was: Open) remove netty dependency through ZK 3.4.x

[jira] [Updated] (KAFKA-1717) remove netty dependency through ZK 3.4.x

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1717: --- Attachment: kafka-1717.patch remove netty dependency through ZK 3.4.x

[jira] [Commented] (KAFKA-1717) remove netty dependency through ZK 3.4.x

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176965#comment-14176965 ] Jun Rao commented on KAFKA-1717: Created reviewboard

Re: Review Request 26936: Patch for kafka-1717

2014-10-20 Thread Sriharsha Chintalapani
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26936/#review57336 --- Ship it! +1 - Sriharsha Chintalapani On Oct. 20, 2014, 3:16

[jira] [Created] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Evan Huus (JIRA)
Evan Huus created KAFKA-1718: Summary: Message Size Too Large error when only small messages produced with Snappy Key: KAFKA-1718 URL: https://issues.apache.org/jira/browse/KAFKA-1718 Project: Kafka

Re: [DISCUSSION] Message Metadata

2014-10-20 Thread Jun Rao
8. Perhaps I need to see the details of the alternative solution. During the startup of a follower, in general, it's not enough for the follower to just see the latest generation of the current leader, since the follower can be several generations behind. So, if the controlled message only

Re: [DISCUSSION] Message Metadata

2014-10-20 Thread Guozhang Wang
I have updated the wiki page incorporating received comments. We can discuss some more details on: 1. How we want to do audit? Whether we want to have in-built auditing on brokers or even MMs or use an audit consumer to fetch all messages from just brokers. 2. How we can avoid

Re: Review Request 26936: Patch for kafka-1717

2014-10-20 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26936/#review57357 --- Ship it! Ship It! - Neha Narkhede On Oct. 20, 2014, 3:16 p.m.,

[jira] [Commented] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177107#comment-14177107 ] Sriharsha Chintalapani commented on KAFKA-1718: --- [~eapache] kafka uses

[jira] [Commented] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Evan Huus (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177112#comment-14177112 ] Evan Huus commented on KAFKA-1718: -- LogOverhead is only 12 bytes; none of the values I

[jira] [Created] (KAFKA-1719) Make mirror maker exit when one consumer/producer thread exits.

2014-10-20 Thread Jiangjie Qin (JIRA)
Jiangjie Qin created KAFKA-1719: --- Summary: Make mirror maker exit when one consumer/producer thread exits. Key: KAFKA-1719 URL: https://issues.apache.org/jira/browse/KAFKA-1719 Project: Kafka

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Vladimir Tretyakov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177149#comment-14177149 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, fuf:), created new patch (for

[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Vladimir Tretyakov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch Stop using dashes AND

[jira] [Updated] (KAFKA-1717) remove netty dependency through ZK 3.4.x

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1717: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the reviews. Committed to trunk

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177250#comment-14177250 ] Jun Rao commented on KAFKA-1481: Thanks for the patch. It doesn't seem to apply. Could you

[jira] [Commented] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177324#comment-14177324 ] Sriharsha Chintalapani commented on KAFKA-1718: --- [~eapache] there is

[Java New Producer] Snappy NPE Issue

2014-10-20 Thread Bhavesh Mistry
Hi Kafka Dev, I am getting following issue with Snappy Library. I checked code for Snappy lib it seems to be fine. Have you guys seen this issue ? 2014-10-20 18:55:21.841 [kafka-producer-network-thread] ERROR org.apache.kafka.clients.producer.internals.Sender - Uncaught error in kafka producer

[jira] [Comment Edited] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Evan Huus (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177351#comment-14177351 ] Evan Huus edited comment on KAFKA-1718 at 10/20/14 7:28 PM:

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Vladimir Tretyakov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177423#comment-14177423 ] Vladimir Tretyakov commented on KAFKA-1481: --- Hi, thx for quick reply, did

[jira] [Updated] (KAFKA-1555) provide strong consistency with reasonable availability

2014-10-20 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1555: Attachment: KAFKA-1555-DOCS.0.patch Adding documentation patch. Note that this patch is huge since

[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Vladimir Tretyakov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Tretyakov updated KAFKA-1481: -- Attachment: KAFKA-1481_2014-10-20_23-14-35.patch

[jira] [Commented] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177478#comment-14177478 ] Jun Rao commented on KAFKA-1718: Another thing that we do on the broker is to assign a new

[jira] [Commented] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Evan Huus (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177487#comment-14177487 ] Evan Huus commented on KAFKA-1718: -- That sounds plausible. 1. How do I verify if that

[jira] [Comment Edited] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Evan Huus (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177487#comment-14177487 ] Evan Huus edited comment on KAFKA-1718 at 10/20/14 9:23 PM:

[jira] [Commented] (KAFKA-1712) Excessive storage usage on newly added node

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177500#comment-14177500 ] Jun Rao commented on KAFKA-1712: This is being discussed in

[jira] [Resolved] (KAFKA-1514) Update Kafka trunk version number

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1514. Resolution: Fixed Update Kafka trunk version number -

[jira] [Created] (KAFKA-1720) [Renaming / Comments] Delayed Operations

2014-10-20 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-1720: Summary: [Renaming / Comments] Delayed Operations Key: KAFKA-1720 URL: https://issues.apache.org/jira/browse/KAFKA-1720 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-1718) Message Size Too Large error when only small messages produced with Snappy

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177558#comment-14177558 ] Jun Rao commented on KAFKA-1718: In trunk, we actually do a size check before and after

Re: Review Request 26373: Patch for KAFKA-1647

2014-10-20 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26373/#review57490 --- Thanks for the patch. Some comments below.

Re: Review Request 26885: Patch for KAFKA-1642

2014-10-20 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26885/ --- (Updated Oct. 21, 2014, 12:34 a.m.) Review request for kafka. Bugs:

[jira] [Updated] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2014-10-20 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1642: - Attachment: KAFKA-1642_2014-10-20_17:33:57.patch [Java New Producer Kafka Trunk]

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177717#comment-14177717 ] Jun Rao commented on KAFKA-1481: It still doesn't apply. git apply -p0

[jira] [Commented] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2014-10-20 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177723#comment-14177723 ] Ewen Cheslack-Postava commented on KAFKA-1642: -- To summarize the issues fixed

[jira] [Updated] (KAFKA-1698) Validator.ensureValid() only validates default config value

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1698: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch. +1 and committed to

[jira] [Updated] (KAFKA-1711) WARN Property topic is not valid when running console producer

2014-10-20 Thread Joe Crobak (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Crobak updated KAFKA-1711: -- Attachment: KAFKA-1711.patch Here's a patch that fixes the warning. WARN Property topic is not valid

Re: [Java New Producer] Snappy NPE Issue

2014-10-20 Thread Ewen Cheslack-Postava
I took a quick look at this since I noticed the same issue when testing your code for the issues you filed. I think the issue is that you're using all your producers across a thread pool and the snappy library uses ThreadLocal BufferRecyclers. When new Snappy streams are allocated, they may be

[jira] [Commented] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

2014-10-20 Thread Sean Fay (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177822#comment-14177822 ] Sean Fay commented on KAFKA-1716: - This happens infrequently at system shutdown. I don't

[jira] [Created] (KAFKA-1721) Snappy compressor is not thread safe

2014-10-20 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-1721: Summary: Snappy compressor is not thread safe Key: KAFKA-1721 URL: https://issues.apache.org/jira/browse/KAFKA-1721 Project: Kafka Issue

Re: [Java New Producer] Snappy NPE Issue

2014-10-20 Thread Ewen Cheslack-Postava
Also, filed https://issues.apache.org/jira/browse/KAFKA-1721 for this since it either requires an updated version of the upstream library, a workaround by us, or at a bare minimum clear documentation of the issue. On Mon, Oct 20, 2014, at 06:23 PM, Ewen Cheslack-Postava wrote: I took a quick

[jira] [Updated] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1642: - Reviewer: Jay Kreps [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network

[jira] [Updated] (KAFKA-1711) WARN Property topic is not valid when running console producer

2014-10-20 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1711: - Reviewer: Jun Rao WARN Property topic is not valid when running console producer

Sending Same Message to Two Topics on Same Broker Cluster

2014-10-20 Thread Bhavesh Mistry
Hi Kakfa Team, I would like to send a single message to multiple topics (two for now) without re-transmitting the message from producer to brokers. Is this possible? Both Producers Scala and Java does not allow this. I do not have to do this all the time only based on application condition.

[jira] [Updated] (KAFKA-1634) Improve semantics of timestamp in OffsetCommitRequests and update documentation

2014-10-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1634: --- Fix Version/s: (was: 0.8.2) 0.8.3 Moving this to 0.8.3 since it's easier to fix after

Re: Sending Same Message to Two Topics on Same Broker Cluster

2014-10-20 Thread Neha Narkhede
Not really. You need producers to send data to Kafka. On Mon, Oct 20, 2014 at 9:05 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi Kakfa Team, I would like to send a single message to multiple topics (two for now) without re-transmitting the message from producer to brokers. Is