Build failed in Jenkins: Kafka-trunk #432

2015-03-25 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/432/changes Changes: [joe.stein] KAFKA-1856 Add PreCommit Patch Testing patch by Ashish K Singh reviewed by Gwen Shapira -- [...truncated 1576 lines...] at

[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work

2015-03-25 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379381#comment-14379381 ] Onur Karaman commented on KAFKA-2046: - I haven't been able to reproduce the issue

[jira] [Updated] (KAFKA-1856) Add PreCommit Patch Testing

2015-03-25 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1856: - Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available) This

Build failed in Jenkins: KafkaPreCommit #44

2015-03-25 Thread Apache Jenkins Server
See https://builds.apache.org/job/KafkaPreCommit/44/changes Changes: [joe.stein] KAFKA-1856 Add PreCommit Patch Testing patch by Ashish K Singh reviewed by Gwen Shapira -- [...truncated 2179 lines...] kafka.javaapi.message.ByteBufferMessageSetTest

[jira] [Updated] (KAFKA-1501) transient unit tests failures due to port already in use

2015-03-25 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1501: - Attachment: KAFKA-1501_2015-03-25_00:44:50.patch transient unit tests failures

Re: Review Request 31806: Patch for KAFKA-1501

2015-03-25 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31806/ --- (Updated March 25, 2015, 7:45 a.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-1501) transient unit tests failures due to port already in use

2015-03-25 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379451#comment-14379451 ] Ewen Cheslack-Postava commented on KAFKA-1501: -- [~guozhang] Sorry, seems I

[jira] [Commented] (KAFKA-2038) Unable to restart brokers after it went down with no space left on disk

2015-03-25 Thread K Zakee (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379515#comment-14379515 ] K Zakee commented on KAFKA-2038: On Mar 21, 2015, at 11:50 AM, svante karlsson s...@csi.se

Re: Review Request 31806: Patch for KAFKA-1501

2015-03-25 Thread Ewen Cheslack-Postava
On March 12, 2015, 1:04 p.m., Guozhang Wang wrote: core/src/test/scala/unit/kafka/utils/TestUtils.scala, line 67 https://reviews.apache.org/r/31806/diff/2/?file=889521#file889521line67 The name is a bit misleading since the port value is actually fixed. Maybe we can renamed it to

[jira] [Commented] (KAFKA-1856) Add PreCommit Patch Testing

2015-03-25 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379900#comment-14379900 ] Ashish K Singh commented on KAFKA-1856: --- Thanks for helping out [~charmalloc]! Add

[jira] [Commented] (KAFKA-1982) change kafka.examples.Producer to use the new java producer

2015-03-25 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379898#comment-14379898 ] Ashish K Singh commented on KAFKA-1982: --- Pinging [~junrao] again :) change

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-25 Thread Jay Kreps
+1 -Jay On Tue, Mar 24, 2015 at 2:43 PM, Guozhang Wang wangg...@gmail.com wrote: +1. On Tue, Mar 24, 2015 at 2:15 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: https://cwiki.apache.org/confluence/display/KAFKA/KIP-15+-+Add+a+close+method+with+a+timeout+in+the+producer As a

Re: Review Request 30809: Patch for KAFKA-1888

2015-03-25 Thread Ashish Singh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30809/#review77732 --- bin/kafka-run-class.sh

[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation

2015-03-25 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379959#comment-14379959 ] Ashish K Singh commented on KAFKA-1994: --- [~gwenshap] My apologies for the delay.

Metrics package discussion

2015-03-25 Thread Aditya Auradkar
Hey everyone, Picking up this discussion after yesterdays KIP hangout. For anyone who did not join the meeting, we have 2 different metrics packages being used by the clients (custom package) and the server (codahale). We are discussing whether to migrate the server to the new package. What

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-25 Thread Jay Kreps
Wait, actually, why would the thread block forever? I would understand throwing an error and failing immediately (fail fast) and I would understand logging an error and blocking for the time they specified (since that is what they asked for), but the logging an error and putatively blocking

Re: Review Request 31806: Patch for KAFKA-1501

2015-03-25 Thread Guozhang Wang
On March 12, 2015, 1:04 p.m., Guozhang Wang wrote: core/src/test/scala/unit/kafka/utils/TestUtils.scala, line 67 https://reviews.apache.org/r/31806/diff/2/?file=889521#file889521line67 The name is a bit misleading since the port value is actually fixed. Maybe we can renamed it to

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-25 Thread Jiangjie Qin
Hi Jay, The reason we keep the blocking behavior if close() or close(timeout) is called from callback is discussed in another thread. I copy/paste the message here: It looks that the problem we want to solve and the purpose we want to achieve is: If user uses close() in callback, we want to let

Re: Review Request 31850: Patch for KAFKA-1660

2015-03-25 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31850/ --- (Updated March 25, 2015, 5:55 p.m.) Review request for kafka. Bugs:

[jira] [Updated] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2015-03-25 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1660: Attachment: KAFKA-1660_2015-03-25_10:55:42.patch Ability to call close() with a timeout on the

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2015-03-25 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380359#comment-14380359 ] Jiangjie Qin commented on KAFKA-1660: - Updated reviewboard

Re: MirrorMaker improvements

2015-03-25 Thread Jun Rao
To amortize the long RTT across data centers, you can tune the TCP window size by configuring a larger socket.receive.buffer.bytes in the consumer. For the last one, it seems that you want identical mirroring. The tricky thing is to figure out how to avoid duplicates when there is a failure. We

Re: MirrorMaker improvements

2015-03-25 Thread Jiangjie Qin
Hi Jun, I think bumping up socket.receive.buffer.bytes will help but might not be sufficient. There are actually two related factors here: 1. Pipelining TCP packets when send a single request/response. 2. Pipelining multiple requests/responses Bumping up socket.receive.buffer.bytes help with the

Re: Review Request 32440: Patch for KAFKA-2043

2015-03-25 Thread Grant Henke
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32440/ --- (Updated March 25, 2015, 6:29 p.m.) Review request for kafka. Bugs:

Re: Review Request 31816: Fix for KAFKA-527

2015-03-25 Thread Guozhang Wang
On March 21, 2015, 1:42 a.m., Jun Rao wrote: core/src/main/scala/kafka/message/ByteBufferMessageSet.scala, line 208 https://reviews.apache.org/r/31816/diff/1/?file=888022#file888022line208 Can we just use wrapperMessage directly? I think we can, as we are just accessing its

Plan of Controlled Shutdown

2015-03-25 Thread Mingjie Lai
Hi. I've been trying to figure out the best way to do kafka broker rolling restart, and read the controlled shutdown wiki page: https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.ControlledShutdown However I cannot find this kafka.admin.ShutdownBroker class

Re: kafka system tests

2015-03-25 Thread Gwen Shapira
Thanks for summarizing! I think we are all feeling the pain here and want to make life easier moving forward. The framework discussion is particularly interesting - unfortunately, the link to ducttape is broken at the moment. Gwen On Wed, Mar 25, 2015 at 11:46 AM, Geoffrey Anderson

Re: Review Request 31742: Patch for KAFKA-527

2015-03-25 Thread Yasuhiro Matsuda
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31742/ --- (Updated March 25, 2015, 7:08 p.m.) Review request for kafka. Bugs:

kafka system tests

2015-03-25 Thread Geoffrey Anderson
Hi dev list, I've been discussing the current state of system tests with Jun and others, and have summarized goals moving forward at: https://cwiki.apache.org/confluence/display/KAFKA/System+Test+Improvements Feedback is welcome! Thanks, Geoff

Re: Review Request 31816: Fix for KAFKA-527

2015-03-25 Thread Guozhang Wang
On March 21, 2015, 1:42 a.m., Jun Rao wrote: core/src/main/scala/kafka/message/ByteBufferMessageSet.scala, lines 241-243 https://reviews.apache.org/r/31816/diff/1/?file=888022#file888022line241 In the normal path, this should only happen when reading the offset. So we should

Re: Plan of Controlled Shutdown

2015-03-25 Thread Harsha
Hi ,        you can enable/disable controlled shutdown using controlled.shutdown.enable in server.properties. This is by default set to “true” in 0.8.2. You can go ahead do a rolling restart  and you don’t need ShutdownBroker command which is removed.  Regarding KAFKA-2029 talks about

[jira] [Updated] (KAFKA-527) Compression support does numerous byte copies

2015-03-25 Thread Yasuhiro Matsuda (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-527: --- Attachment: KAFKA-527_2015-03-25_12:08:00.patch Compression support does numerous byte copies

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-25 Thread Yasuhiro Matsuda (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380547#comment-14380547 ] Yasuhiro Matsuda commented on KAFKA-527: Updated reviewboard

[jira] [Updated] (KAFKA-2043) CompressionType is passed in each RecordAccumulator append

2015-03-25 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KAFKA-2043: --- Attachment: KAFKA-2043_2015-03-25_13:28:52.patch CompressionType is passed in each RecordAccumulator

[jira] [Commented] (KAFKA-2043) CompressionType is passed in each RecordAccumulator append

2015-03-25 Thread Grant Henke (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380433#comment-14380433 ] Grant Henke commented on KAFKA-2043: Updated reviewboard

Re: C++ Client Library -- libkafka-asio

2015-03-25 Thread Daniel Joos
Hi, Thanks for your comment and thank you for adding the library to the wiki page. I haven't done any measurements regarding the performance of the library, yet. This, plus completing the documentation, are the missing parts before version 0.1.0. Of course, compression also needs to be added at

Producer Behavior When one or more Brokers' Disk is Full.

2015-03-25 Thread Bhavesh Mistry
Hello Kafka Community, What is expected behavior on Producer side when one or more Brokers’ disk is full, but have not reached retention period for topics (by size or by time limit). Does producer send data to that particular brokers and/or Producer Queue gets full and always throws Queue

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-25 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380657#comment-14380657 ] Jun Rao commented on KAFKA-527: --- [~ymatsuda], thanks for the patch. +1 and committed to

Re: Review Request 32440: Patch for KAFKA-2043

2015-03-25 Thread Mayuresh Gharat
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32440/#review77787 --- Ship it! - Mayuresh Gharat On March 25, 2015, 6:29 p.m., Grant

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380693#comment-14380693 ] Guozhang Wang commented on KAFKA-527: - Updated reviewboard

Re: Review Request 31816: Fix decompression regarding KAFKA-572

2015-03-25 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31816/ --- (Updated March 25, 2015, 8:26 p.m.) Review request for kafka. Summary

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

2015-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380718#comment-14380718 ] Guozhang Wang commented on KAFKA-2045: -- Agreed, and I remember we did have some

[jira] [Updated] (KAFKA-527) Compression support does numerous byte copies

2015-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-527: Attachment: KAFKA-527_2015-03-25_13:26:36.patch Compression support does numerous byte copies

Re: Review Request 32465: Patch for KAFKA-2047

2015-03-25 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32465/#review77789 --- core/src/main/scala/kafka/tools/MirrorMaker.scala

Build failed in Jenkins: Kafka-trunk #433

2015-03-25 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/433/changes Changes: [junrao] kafka-527; Compression support does numerous byte copies; patched by Yasuhiro Matsuda; reviewed by Guozhang Wang and Jun Rao [wangguoz] KAFKA-2047; Move the stream creation into concurrent mirror maker threads;

Jenkins build is back to normal : KafkaPreCommit #45

2015-03-25 Thread Apache Jenkins Server
See https://builds.apache.org/job/KafkaPreCommit/45/changes

Re: Review Request 31967: Patch for KAFKA-1546

2015-03-25 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31967/ --- (Updated March 25, 2015, 8:27 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2015-03-25 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380696#comment-14380696 ] Aditya A Auradkar commented on KAFKA-1546: -- Updated reviewboard

Re: Review Request 32465: Patch for KAFKA-2047

2015-03-25 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32465/#review77794 --- core/src/main/scala/kafka/tools/MirrorMaker.scala

[jira] [Created] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-03-25 Thread Tim Brooks (JIRA)
Tim Brooks created KAFKA-2050: - Summary: Avoid calling .size() on java.util.ConcurrentLinkedQueue Key: KAFKA-2050 URL: https://issues.apache.org/jira/browse/KAFKA-2050 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-03-25 Thread Tim Brooks (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brooks updated KAFKA-2050: -- Attachment: dont_call_queue_size.patch Avoid calling .size() on java.util.ConcurrentLinkedQueue

Re: [DISCUSS] KIP-11- Authorization design for kafka security

2015-03-25 Thread Parth Brahmbhatt
Hi all, I have modified the KIP to reflect the recent change request from the reviewers. I have been working on the code and I have the server side code for authorization ready. I am now modifying the command line utilities. I would really appreciate if some of the committers can spend sometime

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-25 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380706#comment-14380706 ] Jay Kreps commented on KAFKA-527: - Yeah it would be great to get the [~guozhang]'s patch in

Re: Review Request 32465: Patch for KAFKA-2047

2015-03-25 Thread Jiangjie Qin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32465/ --- (Updated March 25, 2015, 8:48 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-2047) Accelarate consumer rebalance in mirror maker.

2015-03-25 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380737#comment-14380737 ] Jiangjie Qin commented on KAFKA-2047: - Updated reviewboard

Re: Review Request 32465: Patch for KAFKA-2047

2015-03-25 Thread Jiangjie Qin
On March 25, 2015, 8:37 p.m., Guozhang Wang wrote: core/src/main/scala/kafka/tools/MirrorMaker.scala, line 325 https://reviews.apache.org/r/32465/diff/1/?file=904867#file904867line325 Wondering if we call System.exit(-1) directly from one of the MM thread, if we have some

[jira] [Updated] (KAFKA-2047) Accelarate consumer rebalance in mirror maker.

2015-03-25 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-2047: Attachment: KAFKA-2047_2015-03-25_13:48:34.patch Accelarate consumer rebalance in mirror maker.

[jira] [Updated] (KAFKA-2047) Accelarate consumer rebalance in mirror maker.

2015-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2047: - Resolution: Fixed Status: Resolved (was: Patch Available) Accelarate consumer rebalance

[jira] [Commented] (KAFKA-2047) Accelarate consumer rebalance in mirror maker.

2015-03-25 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380758#comment-14380758 ] Guozhang Wang commented on KAFKA-2047: -- Thanks for the patch, +1 and committed to

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32459/#review77805 --- Thanks for the patch. Looks good to me overall. A few comments

[jira] [Updated] (KAFKA-1546) Automate replica lag tuning

2015-03-25 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-1546: - Attachment: KAFKA-1546_2015-03-25_13:27:40.patch Automate replica lag tuning

Re: Review Request 32465: Patch for KAFKA-2047

2015-03-25 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32465/#review77791 --- Ship it! Will make the minor change upon checkin.

[jira] [Commented] (KAFKA-2046) Delete topic still doesn't work

2015-03-25 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380776#comment-14380776 ] Onur Karaman commented on KAFKA-2046: - It looks like I hit a deadlock yesterday

Re: Review Request 31967: Patch for KAFKA-1546

2015-03-25 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31967/#review77799 --- core/src/main/scala/kafka/cluster/Replica.scala

[jira] [Commented] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-03-25 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380809#comment-14380809 ] Sriharsha Chintalapani commented on KAFKA-2050: --- [~timbrooks] changes looks

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
On March 25, 2015, 9:43 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/requests/AbstractRequestResponse.java, lines 46-54 https://reviews.apache.org/r/32459/diff/1/?file=904768#file904768line46 Instead of adding this method here, would it be better to add it

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Jun Rao
On March 25, 2015, 9:43 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/requests/AbstractRequestResponse.java, lines 46-54 https://reviews.apache.org/r/32459/diff/1/?file=904768#file904768line46 Instead of adding this method here, would it be better to add it

[jira] [Updated] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2044: Attachment: KAFKA-2044_2015-03-25_16:53:05.patch Support requests and responses from o.a.k.common

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32459/ --- (Updated March 25, 2015, 11:53 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381056#comment-14381056 ] Gwen Shapira commented on KAFKA-2044: - Updated reviewboard

Re: Review Request 31967: Patch for KAFKA-1546

2015-03-25 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31967/#review77810 --- Thanks for the new patch. A couple of more comments.

[jira] [Updated] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2044: Attachment: KAFKA-2044_2015-03-25_16:48:49.patch Support requests and responses from o.a.k.common

[jira] [Commented] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381051#comment-14381051 ] Gwen Shapira commented on KAFKA-2044: - Updated reviewboard

Re: MirrorMaker improvements

2015-03-25 Thread vlad...@gmail.com
@Guozhang We actually have separate topics depending on the source of the message and the multicast distribution group (the set of destinations). Our topics are named: source_multicast-group. We do not aggregate data but we do static routing based on the destination and the destination set (that

[jira] [Commented] (KAFKA-2038) Unable to restart brokers after it went down with no space left on disk

2015-03-25 Thread K Zakee (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380850#comment-14380850 ] K Zakee commented on KAFKA-2038: Thanks, I was able to start the broker after manually

Re: Review Request 31606: Patch for KAFKA-1416

2015-03-25 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31606/#review77804 --- core/src/test/scala/unit/kafka/integration/FetcherTest.scala

[jira] [Updated] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2044: Attachment: KAFKA-2044_2015-03-25_16:48:01.patch Support requests and responses from o.a.k.common

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32459/ --- (Updated March 25, 2015, 11:48 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381049#comment-14381049 ] Gwen Shapira commented on KAFKA-2044: - Updated reviewboard

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32459/ --- (Updated March 25, 2015, 11:48 p.m.) Review request for kafka. Bugs:

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32459/ --- (Updated March 26, 2015, 1:49 a.m.) Review request for kafka. Bugs:

[jira] [Updated] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2044: Attachment: KAFKA-2044_2015-03-25_18:49:24.patch Support requests and responses from o.a.k.common

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
On March 26, 2015, 12:47 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java, line 53 https://reviews.apache.org/r/32459/diff/4/?file=906206#file906206line53 Cluster may not serialize properly with those nulls in PartitionInfo. Gwen

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
On March 26, 2015, 12:47 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java, line 53 https://reviews.apache.org/r/32459/diff/4/?file=906206#file906206line53 Cluster may not serialize properly with those nulls in PartitionInfo. Gwen

[jira] [Commented] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-03-25 Thread Tim Brooks (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381251#comment-14381251 ] Tim Brooks commented on KAFKA-2050: --- Okay. I submitted it. I had some trouble. But I

[jira] [Commented] (KAFKA-1293) Mirror maker housecleaning

2015-03-25 Thread Matt Warhaftig (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381136#comment-14381136 ] Matt Warhaftig commented on KAFKA-1293: --- Thanks for pointing out KIP-14

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
On March 26, 2015, 12:47 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java, line 53 https://reviews.apache.org/r/32459/diff/4/?file=906206#file906206line53 Cluster may not serialize properly with those nulls in PartitionInfo. Oh

[jira] [Commented] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381206#comment-14381206 ] Gwen Shapira commented on KAFKA-2044: - Updated reviewboard

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Jun Rao
On March 26, 2015, 12:47 a.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java, line 53 https://reviews.apache.org/r/32459/diff/4/?file=906206#file906206line53 Cluster may not serialize properly with those nulls in PartitionInfo. Gwen

[jira] [Updated] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-03-25 Thread Tim Brooks (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brooks updated KAFKA-2050: -- Attachment: KAFKA-2050.patch Avoid calling .size() on java.util.ConcurrentLinkedQueue

[jira] [Commented] (KAFKA-2050) Avoid calling .size() on java.util.ConcurrentLinkedQueue

2015-03-25 Thread Tim Brooks (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381250#comment-14381250 ] Tim Brooks commented on KAFKA-2050: --- Created reviewboard

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32459/#review77832 --- Thanks for the new patch. A few comments below. Perhaps we can add

[jira] [Commented] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381232#comment-14381232 ] Gwen Shapira commented on KAFKA-2044: - Updated reviewboard

Re: Review Request 32459: Patch for KAFKA-2044

2015-03-25 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32459/ --- (Updated March 26, 2015, 2:20 a.m.) Review request for kafka. Bugs:

[jira] [Updated] (KAFKA-2044) Support requests and responses from o.a.k.common in KafkaApis

2015-03-25 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2044: Attachment: KAFKA-2044_2015-03-25_19:20:16.patch Support requests and responses from o.a.k.common

Review Request 32519: Patch for KAFKA-2050

2015-03-25 Thread Tim Brooks
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32519/ --- Review request for kafka. Bugs: KAFKA-2050

Re: Plan of Controlled Shutdown

2015-03-25 Thread Guozhang Wang
Mingjie, Controlled shutdown has been fully implemented and is regularly exercised at LinkedIn for version upgrades / rolling bounce tests, etc. As Harsha said you just need to turn on the config in the brokers, and normal shutting down command like control-c will trigger the controlled shutdown.

Re: [jira] [Commented] (KAFKA-2038) Unable to restart brokers after it went down with no space left on disk

2015-03-25 Thread Clark Haskins
Today the Kafka broker does not have any tools for moving data between volumes on a single broker. Also it doesn't do any fancy calculation to determine where to place new partitions, it simply does round-robin placement across the disks. The way we tackle this at LinkedIn is to run with a single

Re: [VOTE] KIP-15 add a close with timeout to new producer

2015-03-25 Thread Neha Narkhede
We have agreed that we will have an error log to inform user about this mis-usage. The options differ in the way how we can force user to take a look at that error log. Since we have to detect the problem in order to log an appropriate error message, we have a way to tell if the user is

Re: Metrics package discussion

2015-03-25 Thread Neha Narkhede
Aditya, If we are doing a deep dive, one of the things to investigate would be memory/GC performance. IIRC, when I was looking into codahale at LinkedIn, I remember it having quite a few memory management and GC issues while using histograms. In comparison, histograms in the new metrics package

  1   2   >