Re: Review Request 29091: Patch for KAFKA-1646

2014-12-16 Thread Qianlin Xia
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:13 a.m.) Review request for kafka. Bugs:

Re: Review Request 29091: Patch for KAFKA-1646

2014-12-16 Thread Qianlin Xia
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:18 a.m.) Review request for kafka. Bugs:

Re: Review Request 29091: Patch for KAFKA-1646

2014-12-16 Thread Qianlin Xia
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:22 a.m.) Review request for kafka. Bugs:

Re: Review Request 29091: Patch for KAFKA-1646

2014-12-16 Thread Qianlin Xia
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/ --- (Updated Dec. 16, 2014, 8:30 a.m.) Review request for kafka. Bugs:

[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows

2014-12-16 Thread Qianlin Xia (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qianlin Xia updated KAFKA-1646: --- Attachment: KAFKA-1646_20141216_163008.patch Improve consumer read performance for Windows

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2014-12-16 Thread Qianlin Xia (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247936#comment-14247936 ] Qianlin Xia commented on KAFKA-1646: Updated reviewboard

Optimistic locking

2014-12-16 Thread Daniel Schierbeck
I'm trying to design a system that uses Kafka as its primary data store by persisting immutable events into a topic and keeping a secondary index in another data store. The secondary index would store the entities. Each event would pertain to some entity, e.g. a user, and those entities are stored

[jira] [Commented] (KAFKA-1650) Mirror Maker could lose data on unclean shutdown.

2014-12-16 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248407#comment-14248407 ] Jiangjie Qin commented on KAFKA-1650: - Updated reviewboard

Re: Review Request 25995: Patch for KAFKA-1650

2014-12-16 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25995/ --- (Updated Dec. 16, 2014, 4:03 p.m.) Review request for kafka. Bugs:

[jira] [Updated] (KAFKA-1499) Broker-side compression configuration

2014-12-16 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1499: --- Attachment: KAFKA-1499_2014-12-16_22:39:10.patch Broker-side compression configuration

Re: Review Request 24704: Patch for KAFKA-1499

2014-12-16 Thread Manikumar Reddy O
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24704/ --- (Updated Dec. 16, 2014, 5:10 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

2014-12-16 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248520#comment-14248520 ] Manikumar Reddy commented on KAFKA-1499: [~jjkoshy] can you review this patch?

API Annotations

2014-12-16 Thread Gwen Shapira
Hi, Kafka has public APIs in Java and Scala, intended for use by external developers. In addition, Kafka also exposes many public methods that are intended to use within Kafka but are not intended to be called by external developers. Also, some of the external APIs are less stable than others

Re: Review Request 29091: Patch for KAFKA-1646

2014-12-16 Thread Sriharsha Chintalapani
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29091/#review65216 --- Ship it! Overall it looks good to me. - Sriharsha Chintalapani

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2014-12-16 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248597#comment-14248597 ] Sriharsha Chintalapani commented on KAFKA-1646: --- Thanks [~qixia] for the

[jira] [Commented] (KAFKA-1070) Auto-assign node id

2014-12-16 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248599#comment-14248599 ] Sriharsha Chintalapani commented on KAFKA-1070: --- [~nehanarkhede] Can you

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

2014-12-16 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248602#comment-14248602 ] Gwen Shapira commented on KAFKA-1819: - Thanks for pointing the similar issue

Re: API Annotations

2014-12-16 Thread Joe Stein
This is definitely an area we can improve upon, thanks for bringing it up Gwen. We should also decide on structure for how we publish too that has not been something I have been consistent about for each release that should be standard. We can publish to SVN but I always feel like drift happens

Re: Review Request 29030: KAFKA-742: Existing directories under the Kafka data directory without any data cause process to not start

2014-12-16 Thread Ashish Singh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29030/ --- (Updated Dec. 16, 2014, 6:27 p.m.) Review request for kafka. Changes ---

[jira] [Updated] (KAFKA-742) Existing directories under the Kafka data directory without any data cause process to not start

2014-12-16 Thread Ashish Kumar Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated KAFKA-742: - Attachment: KAFKA-742.1.patch Makes sense. Updated patch. Existing directories under the

Re: Review Request 24704: Patch for KAFKA-1499

2014-12-16 Thread Jonathan Creasy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24704/#review65221 --- core/src/main/scala/kafka/log/Log.scala

Re: Review Request 24704: Patch for KAFKA-1499

2014-12-16 Thread Jonathan Creasy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24704/#review65223 --- Ship it! Ship It! - Jonathan Creasy On Dec. 16, 2014, 5:10

Re: [DISCUSSION] adding the serializer api back to the new java producer

2014-12-16 Thread Jun Rao
Joel, With a byte array interface, of course there is nothing that one can't do. However, the real question is that whether we want to encourage people to use it this way or not. Being able to flow just bytes is definitely easier to get started. That's why many early adopters choose to do it that

Re: API Annotations

2014-12-16 Thread Jay Kreps
Hey Gwen, We discussed this a bit about this when starting on the new clients. We were super sloppy about this in initial Kafka development--single jar, no real differentiation between public and private apis. The plan was something like the following: 1. Start to consider this with the new

[jira] [Commented] (KAFKA-1806) broker can still expose uncommitted data to a consumer

2014-12-16 Thread Evan Huus (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248764#comment-14248764 ] Evan Huus commented on KAFKA-1806: -- Sarama client maintainer here (via

[jira] [Commented] (KAFKA-1806) broker can still expose uncommitted data to a consumer

2014-12-16 Thread lokesh Birla (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248993#comment-14248993 ] lokesh Birla commented on KAFKA-1806: - This problem occurs multiple times in

[jira] [Commented] (KAFKA-1479) Logs filling up while Kafka ReplicaFetcherThread tries to retrieve partition info for deleted topics

2014-12-16 Thread Arup Malakar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249071#comment-14249071 ] Arup Malakar commented on KAFKA-1479: - For people who may stumble upon this JIRA, the

[jira] [Commented] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

2014-12-16 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249131#comment-14249131 ] Ewen Cheslack-Postava commented on KAFKA-1788: -- [~bpot] that sounds right,

[jira] [Created] (KAFKA-1822) Add echo request

2014-12-16 Thread Gwen Shapira (JIRA)
Gwen Shapira created KAFKA-1822: --- Summary: Add echo request Key: KAFKA-1822 URL: https://issues.apache.org/jira/browse/KAFKA-1822 Project: Kafka Issue Type: Improvement Components:

[jira] [Commented] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

2014-12-16 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249255#comment-14249255 ] Jay Kreps commented on KAFKA-1788: -- Currently the producer supports either blocking or

[jira] [Commented] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

2014-12-16 Thread Bhavesh Mistry (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249266#comment-14249266 ] Bhavesh Mistry commented on KAFKA-1788: --- [~jkreps], Can we just take quick look at

[jira] [Comment Edited] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

2014-12-16 Thread Bhavesh Mistry (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249266#comment-14249266 ] Bhavesh Mistry edited comment on KAFKA-1788 at 12/17/14 1:26 AM:

[jira] [Comment Edited] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

2014-12-16 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249443#comment-14249443 ] Neha Narkhede edited comment on KAFKA-1819 at 12/17/14 4:08 AM:

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

2014-12-16 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249443#comment-14249443 ] Neha Narkhede commented on KAFKA-1819: -- [~jjkoshy] Shouldn't we remove the deleted

Re: Review Request 29030: KAFKA-742: Existing directories under the Kafka data directory without any data cause process to not start

2014-12-16 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29030/#review65287 --- Ship it! Ship It! - Neha Narkhede On Dec. 16, 2014, 6:27 p.m.,

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

2014-12-16 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249451#comment-14249451 ] Gwen Shapira commented on KAFKA-1819: - The recovery checkpoints are currently handled

[jira] [Updated] (KAFKA-742) Existing directories under the Kafka data directory without any data cause process to not start

2014-12-16 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-742: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch,

Re: Review Request 23702: Patch for KAFKA-1070

2014-12-16 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/#review65289 --- Overall, this looks good. How about adding some unit tests before

Re: Review Request 23702: Patch for KAFKA-1070

2014-12-16 Thread Neha Narkhede
On Dec. 17, 2014, 4:58 a.m., Neha Narkhede wrote: Overall, this looks good. How about adding some unit tests before checking it in? Please ignore this. I was looking at only part of the patch. - Neha --- This is an automatically

Re: Review Request 23702: Patch for KAFKA-1070

2014-12-16 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/#review65293 --- core/src/main/scala/kafka/common/GenerateBrokerIdException.scala

Build failed in Jenkins: Kafka-trunk #355

2014-12-16 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/355/changes Changes: [neha.narkhede] KAFKA-742: Existing directories under the Kafka data directory without any data cause process to not start; reviewed by Neha Narkhede -- [...truncated 1516 lines...]

[jira] [Commented] (KAFKA-1070) Auto-assign node id

2014-12-16 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249492#comment-14249492 ] Neha Narkhede commented on KAFKA-1070: -- Thanks for the updated patch [~sriharsha].

Re: Review Request 29072: rebased for 0.8.2 branch

2014-12-16 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29072/#review65296 --- I found the same review comments I had on trunk. Can you update

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

2014-12-16 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249530#comment-14249530 ] Joel Koshy commented on KAFKA-1819: --- [~gwenshap] that's right - that's what I meant by

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

2014-12-16 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249536#comment-14249536 ] Joel Koshy commented on KAFKA-1499: --- Thank you for the ping, and sorry about the delayed

Re: [DISCUSSION] adding the serializer api back to the new java producer

2014-12-16 Thread Joel Koshy
Jun, Thanks for summarizing this - it helps confirm for me that I did not misunderstand anything in this thread so far; and that I disagree with the premise that the steps in using the current byte-oriented API is cumbersome or inflexible. It involves instantiating the K-V serializers in code (as