[jira] [Updated] (KAFKA-1414) Speedup broker startup after hard reset

2014-07-23 Thread Anton Karamanov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Karamanov updated KAFKA-1414: --- Attachment: KAFKA-1414-rev4.patch I've updated the patch. * replaced {{log.io.parallelism}}

[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset

2014-07-23 Thread Anton Karamanov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071573#comment-14071573 ] Anton Karamanov commented on KAFKA-1414: [~junrao], yes, caches were dropped

[jira] [Updated] (KAFKA-1555) provide strong consistency with reasonable availability

2014-07-23 Thread Jiang Wu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Wu updated KAFKA-1555: Description: In a mission critical application, we expect a kafka cluster with 3 brokers can satisfy two

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

2014-07-23 Thread Nicolae Marasoiu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071718#comment-14071718 ] Nicolae Marasoiu commented on KAFKA-1282: - [~junrao] You agree with the approach,

[jira] [Commented] (KAFKA-1555) provide strong consistency with reasonable availability

2014-07-23 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071792#comment-14071792 ] Jun Rao commented on KAFKA-1555: In case 3.1, when C restarts, the protocol is that C can

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

2014-07-23 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071803#comment-14071803 ] Jun Rao commented on KAFKA-1282: Yes. Thanks for picking it up. Disconnect idle socket

Re: Review Request 23705: Addressing Jun's comments

2014-07-23 Thread Manikumar Reddy O
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23705/ --- (Updated July 23, 2014, 2:55 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages

2014-07-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071812#comment-14071812 ] Manikumar Reddy commented on KAFKA-1192: Updated reviewboard

[jira] [Updated] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages

2014-07-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1192: --- Attachment: KAFKA-1192_2014-07-23_20:23:08.patch Enable DumpLogSegments tool to deserialize

Re: Review Request 23705: Addressing Jun's comments

2014-07-23 Thread Manikumar Reddy O
On July 21, 2014, 6:16 p.m., Guozhang Wang wrote: core/src/main/scala/kafka/tools/DumpLogSegments.scala, line 165 https://reviews.apache.org/r/23705/diff/3/?file=636778#file636778line165 Would key also be null possibly? Manikumar Reddy O wrote: This line is surrounded by

[jira] [Commented] (KAFKA-1555) provide strong consistency with reasonable availability

2014-07-23 Thread Robert Withers (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071886#comment-14071886 ] Robert Withers commented on KAFKA-1555: --- In military satellite communications, when

Re: Review Request 23705: Addressing Jun's comments

2014-07-23 Thread Guozhang Wang
On July 21, 2014, 6:16 p.m., Guozhang Wang wrote: core/src/main/scala/kafka/tools/DumpLogSegments.scala, line 165 https://reviews.apache.org/r/23705/diff/3/?file=636778#file636778line165 Would key also be null possibly? Manikumar Reddy O wrote: This line is surrounded by

[jira] [Commented] (KAFKA-1150) Fetch on a replicated topic does not return as soon as possible

2014-07-23 Thread Simon Cooper (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071919#comment-14071919 ] Simon Cooper commented on KAFKA-1150: - Yes, looks like that patch fixes the issue.

Review Request 23858: Patch for KAFKA-1544

2014-07-23 Thread Manikumar Reddy O
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23858/ --- Review request for kafka. Bugs: KAFKA-1544

[jira] [Commented] (KAFKA-1544) LogCleaner may take a long time to shutdown

2014-07-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071937#comment-14071937 ] Manikumar Reddy commented on KAFKA-1544: Created reviewboard

[jira] [Updated] (KAFKA-1544) LogCleaner may take a long time to shutdown

2014-07-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1544: --- Attachment: KAFKA-1544.patch LogCleaner may take a long time to shutdown

[jira] [Commented] (KAFKA-1150) Fetch on a replicated topic does not return as soon as possible

2014-07-23 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071949#comment-14071949 ] Guozhang Wang commented on KAFKA-1150: -- Hi Simon, just wondering did you apply the

Re: Review Request 23858: Patch for KAFKA-1544

2014-07-23 Thread Jay Kreps
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23858/#review48532 --- Is this better than using a condition? I was imagining replacing

[jira] [Assigned] (KAFKA-1544) LogCleaner may take a long time to shutdown

2014-07-23 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy reassigned KAFKA-1544: -- Assignee: Manikumar Reddy LogCleaner may take a long time to shutdown

Re: Review Request 23858: Patch for KAFKA-1544

2014-07-23 Thread Jay Kreps
On July 23, 2014, 4:56 p.m., Jay Kreps wrote: Is this better than using a condition? I was imagining replacing sleep with shutdownCondition.await(backoffMs, TimeUnit.MILLISECONDS) and then in shutdown() we call shutdownCondition.signal(). Actually a condition isn't quite right as there

question about message sets and produce requests/responses

2014-07-23 Thread Dave Peterson
Hello, I have a question about produce requests and responses. Suppose I create a produce request consisting of a single topic T which contains two message sets S1 and S2, with S1 preceding S2. In other words, the request looks like this: ( request header ( T ( S1, S2 ) ) ) Is it possible

[jira] [Updated] (KAFKA-1510) Force offset commits when migrating consumer offsets from zookeeper to kafka

2014-07-23 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1510: -- Summary: Force offset commits when migrating consumer offsets from zookeeper to kafka (was: Force

[jira] [Commented] (KAFKA-1510) Force offset commits when migrating consumer offsets from zookeeper to kafka

2014-07-23 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072437#comment-14072437 ] Joel Koshy commented on KAFKA-1510: --- [~nmarasoiu] do you think you will be able to take

Re: Review Request 23655: Patch for KAFKA-687

2014-07-23 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23655/#review48454 --- core/src/main/scala/kafka/consumer/ConsumerConfig.scala

Re: question about message sets and produce requests/responses

2014-07-23 Thread Joel Koshy
Not sure if I followed the question correctly, but S1 and S2 would be message-sets to different partitions of T - the producer request object contains a map of topic-partition - messageset). So each partition would get its own ack. The response object is just a map of partition - response status

Re: question about message sets and produce requests/responses

2014-07-23 Thread Dave Peterson
Ok, that's all I need to know. Thanks! On Wed, Jul 23, 2014 at 4:33 PM, Joel Koshy jjkosh...@gmail.com wrote: Not sure if I followed the question correctly, but S1 and S2 would be message-sets to different partitions of T - the producer request object contains a map of topic-partition -

Re: question about message sets and produce requests/responses

2014-07-23 Thread Joel Koshy
Also, to be clear, by ack I mean an error code as described here: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse On Wed, Jul 23, 2014 at 04:33:29PM -0700, Joel Koshy wrote: Not sure if I followed the question correctly,

Re: Review Request 23339: Patch for KAFKA-1507

2014-07-23 Thread Sriharsha Chintalapani
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23339/ --- (Updated July 24, 2014, 12:07 a.m.) Review request for kafka. Bugs:

[jira] [Updated] (KAFKA-1507) Using GetOffsetShell against non-existent topic creates the topic unintentionally

2014-07-23 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1507: -- Attachment: KAFKA-1507_2014-07-23_17:07:20.patch Using GetOffsetShell against

[jira] [Commented] (KAFKA-1507) Using GetOffsetShell against non-existent topic creates the topic unintentionally

2014-07-23 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072603#comment-14072603 ] Sriharsha Chintalapani commented on KAFKA-1507: --- Updated reviewboard

[jira] [Resolved] (KAFKA-1192) Enable DumpLogSegments tool to deserialize messages

2014-07-23 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1192. Resolution: Fixed Fix Version/s: 0.8.2 Assignee: Manikumar Reddy (was: Guozhang Wang)

Re: Review Request 23339: Patch for KAFKA-1507

2014-07-23 Thread Sriharsha Chintalapani
On July 22, 2014, 6:52 p.m., Jun Rao wrote: core/src/main/scala/kafka/client/ClientUtils.scala, lines 86-87 https://reviews.apache.org/r/23339/diff/2/?file=639404#file639404line86 Perhaps this method can be named better since it may not just be for the consumer. Changed it to

Re: Review Request 23339: Patch for KAFKA-1507

2014-07-23 Thread Sriharsha Chintalapani
On July 22, 2014, 6:52 p.m., Jun Rao wrote: Thanks for the patch and doing the rebasing. Some comments below. Thanks for the review. I've noticed NetworkClient.java always uses the latestVersion of an api. There doesn't look like the way I can send specific version of a request since the

[jira] [Updated] (KAFKA-1549) dead brokers coming in the TopicMetadataResponse

2014-07-23 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1549: - Assignee: nicu marasoiu (was: Jun Rao) dead brokers coming in the TopicMetadataResponse

Build failed in Jenkins: Kafka-trunk #231

2014-07-23 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/231/changes Changes: [junrao] kafka-1192; Enable DumpLogSegments tool to deserialize messages; patched by Manikumar Reddy; reviewed by Guozhang Wang and Jun Rao -- [...truncated 534 lines...]

Re: Review Request 23702: Patch for KAFKA-1070

2014-07-23 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/#review48598 --- core/src/main/scala/kafka/server/KafkaServer.scala

[jira] [Commented] (KAFKA-1421) Error: Could not find or load main class kafka.perf.SimpleConsumerPerformance

2014-07-23 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072677#comment-14072677 ] Jun Rao commented on KAFKA-1421: You will need to include kafka-perf jar too. Error:

Re: Review Request 23702: Patch for KAFKA-1070

2014-07-23 Thread Sriharsha Chintalapani
On July 24, 2014, 1:21 a.m., Neha Narkhede wrote: core/src/main/scala/kafka/utils/ZkUtils.scala, line 718 https://reviews.apache.org/r/23702/diff/3/?file=639483#file639483line718 Shouldn't we be using createPersistentSequential? if I used createPersistentSequential it keeps

Re: Review Request 23702: Patch for KAFKA-1070

2014-07-23 Thread Sriharsha Chintalapani
On July 24, 2014, 1:21 a.m., Neha Narkhede wrote: core/src/main/scala/kafka/server/KafkaServer.scala, line 96 https://reviews.apache.org/r/23702/diff/3/?file=639482#file639482line96 any reason you want to move this here from the beginning of the class? if the user didn't provide a

Re: Review Request 23702: Patch for KAFKA-1070

2014-07-23 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/#review48608 --- core/src/main/scala/kafka/server/KafkaServer.scala

[jira] [Commented] (KAFKA-1555) provide strong consistency with reasonable availability

2014-07-23 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072690#comment-14072690 ] Jun Rao commented on KAFKA-1555: Rob, Is that any different from just running with a

Re: Review Request 23702: Patch for KAFKA-1070

2014-07-23 Thread Sriharsha Chintalapani
On July 24, 2014, 1:41 a.m., Jun Rao wrote: core/src/main/scala/kafka/server/KafkaServer.scala, lines 370-371 https://reviews.apache.org/r/23702/diff/3/?file=639482#file639482line370 This probably only needs to be done after line 365? I was thinking of a case where a user has one

Re: [jira] [Commented] (KAFKA-1555) provide strong consistency with reasonable availability

2014-07-23 Thread Robert Withers
Hi Jun, Yes, that's what I am thinking. It allows maintaining a pool of offline, but current and consistent replica shadows, ready to be flipped into ISR. Due to their being out of ISR prevents them being counted in quorum, yet ready to go, so no impact to the producers. Looking at it

[jira] [Commented] (KAFKA-1555) provide strong consistency with reasonable availability

2014-07-23 Thread Robert Withers (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072700#comment-14072700 ] Robert Withers commented on KAFKA-1555: --- Hi Jun, Yes, that's what I am thinking. It

[jira] [Commented] (KAFKA-1555) provide strong consistency with reasonable availability

2014-07-23 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072756#comment-14072756 ] Jun Rao commented on KAFKA-1555: Rob, In order for those shadow replicas to be ready to

Re: Review Request 23474: Patch for KAFKA-1483

2014-07-23 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23474/#review48624 --- Ship it! Ship It! - Neha Narkhede On July 16, 2014, 6:07 p.m.,