Re: Review Request 29467: Patch for KAFKA-1660
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74833 --- Ship it! LGTM. - Jiangjie Qin On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
Re: How to get a JIRA assigned
Filed INFRA-9219 for this. On Mon, Feb 23, 2015 at 5:29 AM, Joe Stein joe.st...@stealth.ly wrote: Jay, I thought it was the same issue like with confluence and comments and why we have to grant rights for that. Bots coming and reassigning everything to them or something in JIRA. We could ask/open a ticket with INFRA, if nothing else maybe help come up with a different way to solve it. ~ Joestein On Sun, Feb 22, 2015 at 7:31 PM, Jay Kreps jay.kr...@gmail.com wrote: Anyone know if there a way to turn this off? Is it possible to configure JIRA to let anyone assign them? Unlike the other lockdown stuff which prevents spam this doesn't seem like it could be a spam vector and it would be awesome to make it easier for people. -Jay On Sun, Feb 22, 2015 at 3:43 PM, Guozhang Wang wangg...@gmail.com wrote: Hi Jonathan, You need to be added to the contributor list before can be assigned to jiras, and only committers can do that for you. I have just add you to the list so you should be able to assign yourself now. Guozhang On Sun, Feb 22, 2015 at 3:08 PM, Jonathan Rafalski jonathan.rafal...@gmail.com wrote: Hello, I was wondering if there are any rights to be able to assign JIRA tickets to myself? I found what I think is a bug while working on 1679 so I opened a ticket and was going to assign a review board for both with my solution but now some else has attempted a patch. Just want to be able to assign a ticket to me so time isn't wasted. If it is something that I need to be granted after submitting a few patches that are accepted can someone at least assign 1679 and 1972 to me so nobody else attempts to work while I am? Thanks! Jonathan. Sent from my iPhone -- -- Guozhang -- -- Guozhang
Re: Review Request 29467: Patch for KAFKA-1660
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74835 --- Could we add some unit tests for this new API as I mentioned in my previous comment? - Guozhang Wang On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
Re: Unit tests in java7 vs java8
Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest modified a bit): JDK 8 Total time: 18 mins 3.649 secs real18m4.091s user0m7.105s sys0m0.426s JDK 7 Total time: 18 mins 55.546 secs real18m55.997s user0m4.157s sys0m0.341s Guozhang On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang guw...@linkedin.com.invalid wrote: I am working on the test handing / NPE / failure issues of ConsumerTest only. I currently run Java 8 and the unit test takes about 10 minutes, I can do time ./gradlew test with both versions and see if there is a clear difference. Guozhang From: Jay Kreps [jay.kr...@gmail.com] Sent: Wednesday, February 25, 2015 4:53 PM To: dev@kafka.apache.org; Guozhang Wang Subject: Re: Unit tests in java7 vs java8 Yeah, hey Guozhang, is that fix part of the larger consumer patch you just posted or is that a separate issue? -Jay On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira gshap...@cloudera.com mailto:gshap...@cloudera.com wrote: The Consumer tests are currently hanging :( I think Guozhang is working on a solution. I'm commenting them out until the problem is resolved... On Wed, Feb 25, 2015 at 4:00 PM, Tong Li liton...@us.ibm.commailto: liton...@us.ibm.com wrote: Gwen, I have not tried Java 8. Still on Java 7, but I always run into the test hung problems (no errors on the screen and the system is completely idle), it may be a different problem. I can recreate that problem every time when I run gradle --daemon testAll, I recall that couple of weeks ago there was one patch saying fixed the problem, but I am still seeing the problem with latest code. What I noticed is that seems tests always stop at one of the ConsumerTest test cases. What puzzled me the most is that it was not always a particular test case. Being very new in this community, I think that error must be something related to my env. Here is my environment: Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and big enough max lock memory, not complaining, just some observations in case you wonder what other developers may face. Thanks. Tong Li OpenStack Kafka Community Development Building 501/B205 liton...@us.ibm.commailto:liton...@us.ibm.com [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they take almost twice From: Gwen Shapira gshap...@cloudera.commailto:gshap...@cloudera.com To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Date: 02/25/2015 03:47 PM Subject: Unit tests in java7 vs java8 -- Hi, Anyone running tests on Java 8? I just noticed that they take almost twice as long to run compared to Java 7 (at least on my box, and with Scala 2.10.4). Anyone else noticed this? Maybe even did some digging on the causes? Gwen -- -- Guozhang
Fwd: patch set 1988
Folks, Do not want to nag you, but wonder if any of you has couple of minutes to review patch set for 1988 again so that I do not have to rebase this so many times. Guozhang already +1ed(thanks Guozhang!) Here are the links for your convenience. The issue https://issues.apache.org/jira/browse/KAFKA-1988 The patch set https://reviews.apache.org/r/31566/diff/ Tong Li OpenStack Kafka Community Development Building 501/B205 liton...@us.ibm.com
Re: Review Request 31591: Patch for KAFKA-1992
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31591/#review74827 --- Ship it! LGTM, just a minor comment. core/src/main/scala/kafka/cluster/Partition.scala https://reviews.apache.org/r/31591/#comment121610 This part seems now serving only logging purpose. If that is the case, can we make it even clearer. For example, print all the acked replicas instead of just a number. - Jiangjie Qin On March 1, 2015, 7:58 a.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31591/ --- (Updated March 1, 2015, 7:58 a.m.) Review request for kafka. Bugs: KAFKA-1992 https://issues.apache.org/jira/browse/KAFKA-1992 Repository: kafka Description --- remove unnecessary requiredAcks parameter and clean up few comments Diffs - core/src/main/scala/kafka/cluster/Partition.scala c4bf48a801007ebe7497077d2018d6dffe1677d4 core/src/main/scala/kafka/server/DelayedProduce.scala 4d763bf05efb24a394662721292fc54d32467969 Diff: https://reviews.apache.org/r/31591/diff/ Testing --- Thanks, Gwen Shapira
[jira] [Created] (KAFKA-1998) Partitions Missing From MetadataResponse
Evan Huus created KAFKA-1998: Summary: Partitions Missing From MetadataResponse Key: KAFKA-1998 URL: https://issues.apache.org/jira/browse/KAFKA-1998 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2.0 Reporter: Evan Huus It is known behaviour that when a partition is entirely offline (it has no leader because all of its replicas are down) then that partition will not be included in the metadata returned by other brokers. For example, if topic foo has 3 partitions, but all replicas of partition 3 are offline, then requesting metadata for foo will only return information about partitions 1 and 2. This means that there is no way to reliably determine the number of partitions for a topic via kafka's metadata API; if I receive information on partitions 1 and 2, I don't know if partition 3 is offline or if it is simply that there are only two partitions total. (You can presumably still ask zookeeper directly, but that is a work-around). This ambiguity, in turn, can lead to a consistency problem with the default partitioner, since that effectively implements `hash(key) mod #partitions`. If a partition goes offline and is removed from the metadata response, then the number of partitions the producer knows about will change (on its next metadata refresh) and the mapping from keys to partitions will also change. Instead of distributing messages among (for example) 3 partitions, and failing to produce to the offline partition, it will distribute *all* messages among the two online partitions. This results in messages being sent to the wrong partition. Since kafka already returns partitions with error messages in many cases (e.g. `LeaderNotAvailable`) I think it makes much more sense and fixes the above partition problem if it would simply return offline partitions as well with the appropriate error (whether that is `LeaderNotAvailable` or it would be better to add an additional error is up to you). CC [~guozhang] (This issue was originally described/discussed on the kafka-users mailing list, in the thread involving https://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3CCAA4pprAZvp2XhdNmy0%2BqVZ1UVdVxmUfz3DDArhGbwP-iiH%2BGyg%40mail.gmail.com%3E) If there are any questions I am happy to clarify, I realize the scenario is somewhat complex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29467: Patch for KAFKA-1660
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74843 --- Actually I spoke too fast... As the flush() has been checked in, we need to take care of the caller thread that are doing a flush when invoking close(). This is a little bit tricky. If we close the producer forcibily when caller thread were doing a flush, we have to notify the caller thread that the flush failed. The simplest way might be letting flush return a boolean value. So we do the following: 1. In RecordAccumulator add a new forceClose(), it sets an forceClosed flag first, then clear up the imcomplete batchset and wake up all the caller threads. 2. In RecordAccumulator.awaitFlushCompletion(), it checks the forceClosed flag to determine whether flush succeeded or not and return the result to KafkaProducer.flush(). 3. KafkaProducer.flush() return this result to caller threads. clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java https://reviews.apache.org/r/29467/#comment121626 We probably need to release the caller threads that are waiting on flush() at this point. - Jiangjie Qin On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
[jira] [Assigned] (KAFKA-1998) Partitions Missing From MetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-1998: -- Assignee: Mayuresh Gharat Partitions Missing From MetadataResponse Key: KAFKA-1998 URL: https://issues.apache.org/jira/browse/KAFKA-1998 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2.0 Reporter: Evan Huus Assignee: Mayuresh Gharat It is known behaviour that when a partition is entirely offline (it has no leader because all of its replicas are down) then that partition will not be included in the metadata returned by other brokers. For example, if topic foo has 3 partitions, but all replicas of partition 3 are offline, then requesting metadata for foo will only return information about partitions 1 and 2. This means that there is no way to reliably determine the number of partitions for a topic via kafka's metadata API; if I receive information on partitions 1 and 2, I don't know if partition 3 is offline or if it is simply that there are only two partitions total. (You can presumably still ask zookeeper directly, but that is a work-around). This ambiguity, in turn, can lead to a consistency problem with the default partitioner, since that effectively implements `hash(key) mod #partitions`. If a partition goes offline and is removed from the metadata response, then the number of partitions the producer knows about will change (on its next metadata refresh) and the mapping from keys to partitions will also change. Instead of distributing messages among (for example) 3 partitions, and failing to produce to the offline partition, it will distribute *all* messages among the two online partitions. This results in messages being sent to the wrong partition. Since kafka already returns partitions with error messages in many cases (e.g. `LeaderNotAvailable`) I think it makes much more sense and fixes the above partition problem if it would simply return offline partitions as well with the appropriate error (whether that is `LeaderNotAvailable` or it would be better to add an additional error is up to you). CC [~guozhang] (This issue was originally described/discussed on the kafka-users mailing list, in the thread involving https://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3CCAA4pprAZvp2XhdNmy0%2BqVZ1UVdVxmUfz3DDArhGbwP-iiH%2BGyg%40mail.gmail.com%3E) If there are any questions I am happy to clarify, I realize the scenario is somewhat complex. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Unit tests in java7 vs java8
I guess its just my machine then. Thanks! On Mon, Mar 2, 2015 at 2:47 PM, Guozhang Wang wangg...@gmail.com wrote: Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest modified a bit): JDK 8 Total time: 18 mins 3.649 secs real18m4.091s user0m7.105s sys0m0.426s JDK 7 Total time: 18 mins 55.546 secs real18m55.997s user0m4.157s sys0m0.341s Guozhang On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang guw...@linkedin.com.invalid wrote: I am working on the test handing / NPE / failure issues of ConsumerTest only. I currently run Java 8 and the unit test takes about 10 minutes, I can do time ./gradlew test with both versions and see if there is a clear difference. Guozhang From: Jay Kreps [jay.kr...@gmail.com] Sent: Wednesday, February 25, 2015 4:53 PM To: dev@kafka.apache.org; Guozhang Wang Subject: Re: Unit tests in java7 vs java8 Yeah, hey Guozhang, is that fix part of the larger consumer patch you just posted or is that a separate issue? -Jay On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira gshap...@cloudera.com mailto:gshap...@cloudera.com wrote: The Consumer tests are currently hanging :( I think Guozhang is working on a solution. I'm commenting them out until the problem is resolved... On Wed, Feb 25, 2015 at 4:00 PM, Tong Li liton...@us.ibm.commailto: liton...@us.ibm.com wrote: Gwen, I have not tried Java 8. Still on Java 7, but I always run into the test hung problems (no errors on the screen and the system is completely idle), it may be a different problem. I can recreate that problem every time when I run gradle --daemon testAll, I recall that couple of weeks ago there was one patch saying fixed the problem, but I am still seeing the problem with latest code. What I noticed is that seems tests always stop at one of the ConsumerTest test cases. What puzzled me the most is that it was not always a particular test case. Being very new in this community, I think that error must be something related to my env. Here is my environment: Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and big enough max lock memory, not complaining, just some observations in case you wonder what other developers may face. Thanks. Tong Li OpenStack Kafka Community Development Building 501/B205 liton...@us.ibm.commailto:liton...@us.ibm.com [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they take almost twice From: Gwen Shapira gshap...@cloudera.commailto:gshap...@cloudera.com To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Date: 02/25/2015 03:47 PM Subject: Unit tests in java7 vs java8 -- Hi, Anyone running tests on Java 8? I just noticed that they take almost twice as long to run compared to Java 7 (at least on my box, and with Scala 2.10.4). Anyone else noticed this? Maybe even did some digging on the causes? Gwen -- -- Guozhang
Re: How to get a JIRA assigned
Thanks Guozhang! On Mon, Mar 2, 2015 at 1:59 PM, Guozhang Wang wangg...@gmail.com wrote: Filed INFRA-9219 for this. On Mon, Feb 23, 2015 at 5:29 AM, Joe Stein joe.st...@stealth.ly wrote: Jay, I thought it was the same issue like with confluence and comments and why we have to grant rights for that. Bots coming and reassigning everything to them or something in JIRA. We could ask/open a ticket with INFRA, if nothing else maybe help come up with a different way to solve it. ~ Joestein On Sun, Feb 22, 2015 at 7:31 PM, Jay Kreps jay.kr...@gmail.com wrote: Anyone know if there a way to turn this off? Is it possible to configure JIRA to let anyone assign them? Unlike the other lockdown stuff which prevents spam this doesn't seem like it could be a spam vector and it would be awesome to make it easier for people. -Jay On Sun, Feb 22, 2015 at 3:43 PM, Guozhang Wang wangg...@gmail.com wrote: Hi Jonathan, You need to be added to the contributor list before can be assigned to jiras, and only committers can do that for you. I have just add you to the list so you should be able to assign yourself now. Guozhang On Sun, Feb 22, 2015 at 3:08 PM, Jonathan Rafalski jonathan.rafal...@gmail.com wrote: Hello, I was wondering if there are any rights to be able to assign JIRA tickets to myself? I found what I think is a bug while working on 1679 so I opened a ticket and was going to assign a review board for both with my solution but now some else has attempted a patch. Just want to be able to assign a ticket to me so time isn't wasted. If it is something that I need to be granted after submitting a few patches that are accepted can someone at least assign 1679 and 1972 to me so nobody else attempts to work while I am? Thanks! Jonathan. Sent from my iPhone -- -- Guozhang -- -- Guozhang -- Thanks, Neha
Re: Unit tests in java7 vs java8
Total time: 14 mins 57.037 secs And I'm running with SSD. On Mon, Mar 2, 2015 at 4:34 PM, Jay Kreps jay.kr...@gmail.com wrote: Wow, 18 mins? I was seeing 8 mins a couple weeks ago and 12 mins now. Anyone know what's up? Not sure if the 12=18 is just because I have SSDs or what. It is really easy to make a small change that adds a few hundred ms of startup or shutdown time and that have that multiply by 500 server start and stops in the test execution. -Jay On Mon, Mar 2, 2015 at 2:47 PM, Guozhang Wang wangg...@gmail.com wrote: Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest modified a bit): JDK 8 Total time: 18 mins 3.649 secs real18m4.091s user0m7.105s sys0m0.426s JDK 7 Total time: 18 mins 55.546 secs real18m55.997s user0m4.157s sys0m0.341s Guozhang On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang guw...@linkedin.com.invalid wrote: I am working on the test handing / NPE / failure issues of ConsumerTest only. I currently run Java 8 and the unit test takes about 10 minutes, I can do time ./gradlew test with both versions and see if there is a clear difference. Guozhang From: Jay Kreps [jay.kr...@gmail.com] Sent: Wednesday, February 25, 2015 4:53 PM To: dev@kafka.apache.org; Guozhang Wang Subject: Re: Unit tests in java7 vs java8 Yeah, hey Guozhang, is that fix part of the larger consumer patch you just posted or is that a separate issue? -Jay On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira gshap...@cloudera.com mailto:gshap...@cloudera.com wrote: The Consumer tests are currently hanging :( I think Guozhang is working on a solution. I'm commenting them out until the problem is resolved... On Wed, Feb 25, 2015 at 4:00 PM, Tong Li liton...@us.ibm.commailto: liton...@us.ibm.com wrote: Gwen, I have not tried Java 8. Still on Java 7, but I always run into the test hung problems (no errors on the screen and the system is completely idle), it may be a different problem. I can recreate that problem every time when I run gradle --daemon testAll, I recall that couple of weeks ago there was one patch saying fixed the problem, but I am still seeing the problem with latest code. What I noticed is that seems tests always stop at one of the ConsumerTest test cases. What puzzled me the most is that it was not always a particular test case. Being very new in this community, I think that error must be something related to my env. Here is my environment: Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and big enough max lock memory, not complaining, just some observations in case you wonder what other developers may face. Thanks. Tong Li OpenStack Kafka Community Development Building 501/B205 liton...@us.ibm.commailto:liton...@us.ibm.com [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they take almost twice From: Gwen Shapira gshap...@cloudera.commailto:gshap...@cloudera.com To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Date: 02/25/2015 03:47 PM Subject: Unit tests in java7 vs java8 -- Hi, Anyone running tests on Java 8? I just noticed that they take almost twice as long to run compared to Java 7 (at least on my box, and with Scala 2.10.4). Anyone else noticed this? Maybe even did some digging on the causes? Gwen -- -- Guozhang
[jira] [Comment Edited] (KAFKA-1910) Refactor KafkaConsumer
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14336985#comment-14336985 ] Guozhang Wang edited comment on KAFKA-1910 at 3/3/15 12:45 AM: --- The uploaded patch contains multiple fixes to the related JIRAs as well as refactoring the new consumer itself. I will summarize them here instead of in the RB: 1. Fix ConsumerTest.testXXXwithBrokerFailure: in RestartDeadBroker we need to call startup() on the old brokers instead of creating new ones as the last approach will case the metadata to be mess up and cause the test to hang (KAFKA-1948). Also make sure the test topic is created with correct replication factor to avoid hanging when the only replica broker was shutdown. Also make the bouncing of the brokers in the background thread so that it will eventually be restarted. 2. Fix ConsumerTest's __consumer_offsets topic: when we call partitionFor() the __consumer_offsets topic may be created with replication as min(offsetTopicRaplicationFactor, aliveBrokers.size), see KAFKA-1864 for details (KAFKA-1975). 3. Add the IllegalGeneration logic in the coordinator as it is important for consumers rebalancing after rediscovering the coordinator, in the current stub it always return OK and hence consumers migrating to the new coordinator will not trigger rebalance (KAFKA-1964). 4. Create the Coodinator and the FetchManager modules as KafkaConsumer internals. Coordinator is responsible for assign partitions (join groups), commit offsets and fetch offsets from coordinator, and FetchManager is responsible for handling fetch request / responses. 4.1 After the refactoring it is easier to detect and fix a bug where response callbacks being triggered multiple times, causing the coordinator NPE (KAFKA-1969). 4.2 Avoid always trying to fetch offsets from coordinator whenever the consumer decides to update fetch positions, introduce a few new variables / APIs in SubscriptionState accordingly. 4.3 Move serializer / de-serializer configs / constructors to AbstractConfig. 4.4 Add missing error handling in commit offset / heartbeat responses. In general I think we should make notes about possible error codes in each of the response type to help coding error handling logic, has filed KAFKA-1985 for that. was (Author: guozhang): The uploaded patch contains multiple fixes to the related JIRAs as well as refactoring the new consumer itself. I will summarize them here instead of in the RB: 1. Fix ConsumerTest.testXXXwithBrokerFailure: in RestartDeadBroker we need to call startup() on the old brokers instead of creating new ones as the last approach will case the metadata to be mess up and cause the test to hang (KAFKA-1948). Also make sure the test topic is created with correct replication factor to avoid hanging when the only replica broker was shutdown. 2. Fix ConsumerTest's __consumer_offsets topic: when we call partitionFor() the __consumer_offsets topic may be created with replication as min(offsetTopicRaplicationFactor, aliveBrokers.size), see KAFKA-1864 for details (KAFKA-1975). 3. Add the IllegalGeneration logic in the coordinator as it is important for consumers rebalancing after rediscovering the coordinator, in the current stub it always return OK and hence consumers migrating to the new coordinator will not trigger rebalance (KAFKA-1964). 4. Create the Coodinator and the FetchManager modules as KafkaConsumer internals. Coordinator is responsible for assign partitions (join groups), commit offsets and fetch offsets from coordinator, and FetchManager is responsible for handling fetch request / responses. 4.1 After the refactoring it is easier to detect and fix a bug where response callbacks being triggered multiple times, causing the coordinator NPE (KAFKA-1969). 4.2 Avoid always trying to fetch offsets from coordinator whenever the consumer decides to update fetch positions, introduce a few new variables / APIs in SubscriptionState accordingly. 4.3 Move serializer / de-serializer configs / constructors to AbstractConfig. 4.4 Add missing error handling in commit offset / heartbeat responses. In general I think we should make notes about possible error codes in each of the response type to help coding error handling logic, has filed KAFKA-1985 for that. Refactor KafkaConsumer -- Key: KAFKA-1910 URL: https://issues.apache.org/jira/browse/KAFKA-1910 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang KafkaConsumer now contains all the logic on the consumer side, making it a very huge class file, better re-factoring it to have multiple layers on top of KafkaClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344463#comment-14344463 ] Grant Henke commented on KAFKA-902: --- A few thoughts on the patch: Should the jitter be added to 'reconnect.backoff.ms' too? Would there ever be a good reason to change the jitter value from 10? Should it be added to the CommonClientConfigs? Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] 0.8.2.1 Candidate 2
+1 from me. Verified quickstart and unit tests. Thanks, Jun On Thu, Feb 26, 2015 at 2:59 PM, Jun Rao j...@confluent.io wrote: This is the second candidate for release of Apache Kafka 0.8.2.1. This fixes 4 critical issue in 0.8.2.0. Release Notes for the 0.8.2.1 release https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/RELEASE_NOTES.html *** Please download, test and vote by Monday, Mar 2, 3pm PT Kafka's KEYS file containing PGP keys we use to sign the release: http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 (SHA256) checksum. * Release artifacts to be voted upon (source and binary): https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/ * Maven artifacts to be voted upon prior to release: https://repository.apache.org/content/groups/staging/ * scala-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/scaladoc/ * java-doc https://people.apache.org/~junrao/kafka-0.8.2.1-candidate2/javadoc/ * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.1 tag https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bd1bfb63ec73c10d08432ac893a23f28281ea021 (git commit ee1267b127f3081db491fa1bf9a287084c324e36) /*** Thanks, Jun
[jira] [Updated] (KAFKA-1910) Refactor KafkaConsumer
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1910: - Attachment: KAFKA-1910.patch Refactor KafkaConsumer -- Key: KAFKA-1910 URL: https://issues.apache.org/jira/browse/KAFKA-1910 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang Attachments: KAFKA-1910.patch KafkaConsumer now contains all the logic on the consumer side, making it a very huge class file, better re-factoring it to have multiple layers on top of KafkaClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1910) Refactor KafkaConsumer
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1910: - Status: Patch Available (was: Open) Refactor KafkaConsumer -- Key: KAFKA-1910 URL: https://issues.apache.org/jira/browse/KAFKA-1910 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang Attachments: KAFKA-1910.patch KafkaConsumer now contains all the logic on the consumer side, making it a very huge class file, better re-factoring it to have multiple layers on top of KafkaClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31650: Drag Coordinator and FetchManager out of KafkaConsumer, fix a bunch of consumer test issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31650/ --- Review request for kafka. Bugs: KAFKA-1910 https://issues.apache.org/jira/browse/KAFKA-1910 Repository: kafka Description --- See comments in KAFKA-1910 Diffs - clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 06fcfe62cc1fe76f58540221698ef076fe150e96 clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 8a3e55aaff7d8c26e56a8407166a4176c1da2644 clients/src/main/java/org/apache/kafka/clients/NetworkClient.java a7fa4a9dfbcfbc4d9e9259630253cbcded158064 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 5fb21001abd77cac839bd724afa04e377a3e82aa clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 67ceb754a52c07143c69b053fe128b9e24060b99 clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchManager.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/internals/Heartbeat.java ee0751e4949120d114202c2299d49612a89b9d97 clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java d41d3068c11d4b5c640467dc0ae1b7c20a8d128c clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 122375c473bf73caf05299b9f5174c6b226ca863 clients/src/main/java/org/apache/kafka/common/network/Selector.java 6baad9366a1975dbaba1786da91efeaa38533319 clients/src/main/java/org/apache/kafka/common/protocol/Errors.java ad2171f5417c93194f5f234bdc7fdd0b8d59a8a8 clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java e67c4c8332cb1dd3d9cde5de687df7760045dfe6 clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java 0057496228feeeccbc0c009a42f5268fa2cb8611 clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java 8c50e9be534c61ecf56106bf2b68cf678ea50d66 clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java 52b1803d8b558c1eeb978ba8821496c7d3c20a6b clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java cfac47a4a05dc8a535595542d93e55237b7d1e93 clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 90f31413d7d80a06c0af359009cc271aa0c67be3 clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java 4d3b9ececee4b4c0b50ba99da2ddbbb15f9cc08d clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java edbed5880dc44fc178737a5e298c106a00f38443 clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java a00dcdf15d1c7bac7228be140647bd7d849deb9b clients/src/test/java/org/apache/kafka/clients/MockClient.java 8f1a7a625e4eeafa44bbf9e5cff987de86c949be core/src/main/scala/kafka/common/ErrorMapping.scala eedc2f5f21dd8755fba891998456351622e17047 core/src/main/scala/kafka/common/NoOffsetsCommittedException.scala PRE-CREATION core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 4cabffeacea09a49913505db19a96a55d58c0909 core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 21790a5059ee00d6610be6f0389445327b88db1d core/src/main/scala/kafka/coordinator/ConsumerRegistry.scala b65c04d0a5d53bf92299d5f67f112be3da3bf77d core/src/main/scala/kafka/coordinator/DelayedHeartbeat.scala b1248e95d8a648b461f604c154879cc95dc7b1cb core/src/main/scala/kafka/coordinator/GroupRegistry.scala 7d17e102235134b6312271c4061abd27d7177f7e core/src/main/scala/kafka/server/KafkaServer.scala 426e522fc9819a0fc0f4e8269033552d716eb066 core/src/test/scala/integration/kafka/api/ConsumerTest.scala PRE-CREATION core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 5650b4a7b950b48af3e272947bfb5e271c4238c9 core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala ba48a636dd0b0ed06646d56bb36aa3d43228604f core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala dc0512b526e914df7e7581b27df18f498da428e2 core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala a2bb8855c3c0586b6b45b53ce534edee31b3bd12 core/src/test/scala/unit/kafka/utils/TestUtils.scala 6ce18076f6b5deb05b51c25be5bed9957e6b4339 Diff: https://reviews.apache.org/r/31650/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344174#comment-14344174 ] Guozhang Wang commented on KAFKA-1910: -- Created reviewboard https://reviews.apache.org/r/31650/diff/ against branch origin/trunk Refactor KafkaConsumer -- Key: KAFKA-1910 URL: https://issues.apache.org/jira/browse/KAFKA-1910 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang Attachments: KAFKA-1910.patch KafkaConsumer now contains all the logic on the consumer side, making it a very huge class file, better re-factoring it to have multiple layers on top of KafkaClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Unit tests in java7 vs java8
Wow, 18 mins? I was seeing 8 mins a couple weeks ago and 12 mins now. Anyone know what's up? Not sure if the 12=18 is just because I have SSDs or what. It is really easy to make a small change that adds a few hundred ms of startup or shutdown time and that have that multiply by 500 server start and stops in the test execution. -Jay On Mon, Mar 2, 2015 at 2:47 PM, Guozhang Wang wangg...@gmail.com wrote: Here are some number I got (this is with KAFKA-1910 patch and ConsumerTest modified a bit): JDK 8 Total time: 18 mins 3.649 secs real18m4.091s user0m7.105s sys0m0.426s JDK 7 Total time: 18 mins 55.546 secs real18m55.997s user0m4.157s sys0m0.341s Guozhang On Wed, Feb 25, 2015 at 5:06 PM, Guozhang Wang guw...@linkedin.com.invalid wrote: I am working on the test handing / NPE / failure issues of ConsumerTest only. I currently run Java 8 and the unit test takes about 10 minutes, I can do time ./gradlew test with both versions and see if there is a clear difference. Guozhang From: Jay Kreps [jay.kr...@gmail.com] Sent: Wednesday, February 25, 2015 4:53 PM To: dev@kafka.apache.org; Guozhang Wang Subject: Re: Unit tests in java7 vs java8 Yeah, hey Guozhang, is that fix part of the larger consumer patch you just posted or is that a separate issue? -Jay On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira gshap...@cloudera.com mailto:gshap...@cloudera.com wrote: The Consumer tests are currently hanging :( I think Guozhang is working on a solution. I'm commenting them out until the problem is resolved... On Wed, Feb 25, 2015 at 4:00 PM, Tong Li liton...@us.ibm.commailto: liton...@us.ibm.com wrote: Gwen, I have not tried Java 8. Still on Java 7, but I always run into the test hung problems (no errors on the screen and the system is completely idle), it may be a different problem. I can recreate that problem every time when I run gradle --daemon testAll, I recall that couple of weeks ago there was one patch saying fixed the problem, but I am still seeing the problem with latest code. What I noticed is that seems tests always stop at one of the ConsumerTest test cases. What puzzled me the most is that it was not always a particular test case. Being very new in this community, I think that error must be something related to my env. Here is my environment: Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and big enough max lock memory, not complaining, just some observations in case you wonder what other developers may face. Thanks. Tong Li OpenStack Kafka Community Development Building 501/B205 liton...@us.ibm.commailto:liton...@us.ibm.com [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I just noticed that they take almost twice From: Gwen Shapira gshap...@cloudera.commailto:gshap...@cloudera.com To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Date: 02/25/2015 03:47 PM Subject: Unit tests in java7 vs java8 -- Hi, Anyone running tests on Java 8? I just noticed that they take almost twice as long to run compared to Java 7 (at least on my box, and with Scala 2.10.4). Anyone else noticed this? Maybe even did some digging on the causes? Gwen -- -- Guozhang
[jira] [Updated] (KAFKA-1882) Create extendable channel interface and default implementations
[ https://issues.apache.org/jira/browse/KAFKA-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1882: - Status: Patch Available (was: Open) Create extendable channel interface and default implementations --- Key: KAFKA-1882 URL: https://issues.apache.org/jira/browse/KAFKA-1882 Project: Kafka Issue Type: Sub-task Components: security Reporter: Gwen Shapira Assignee: Gwen Shapira Priority: Blocker Fix For: 0.8.3 For the security infrastructure, we need an extendible interface to replace SocketChannel. KAFKA-1684 suggests extending SocketChannel itself, but since SocketChannel is part of Java's standard library, the interface changes between different Java versions, so extending it directly can become a compatibility issue. Instead, we can implement a KafkaChannel interface, which will implement connect(), read(), write() and possibly other methods we use. We will replace direct use of SocketChannel in our code with use of KafkaChannel. Different implementations of KafkaChannel will be instantiated based on the port/SecurityProtocol configuration. This patch will provide at least the PLAINTEXT implementation for KafkaChannel. I will validate that the SSL implementation in KAFKA-1684 can be refactored to use a KafkaChannel interface rather than extend SocketChannel directly. However, the patch will not include the SSL channel itself. The interface should also include setters/getters for principal and remote IP, which will be used for the authentication code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1882) Create extendable channel interface and default implementations
[ https://issues.apache.org/jira/browse/KAFKA-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1882: - Priority: Blocker (was: Major) Fix Version/s: 0.8.3 supported in this patch https://issues.apache.org/jira/browse/KAFKA-1809 with PLAINTEXT as the default implementation. The KIP has been accepted too https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs Create extendable channel interface and default implementations --- Key: KAFKA-1882 URL: https://issues.apache.org/jira/browse/KAFKA-1882 Project: Kafka Issue Type: Sub-task Components: security Reporter: Gwen Shapira Assignee: Gwen Shapira Priority: Blocker Fix For: 0.8.3 For the security infrastructure, we need an extendible interface to replace SocketChannel. KAFKA-1684 suggests extending SocketChannel itself, but since SocketChannel is part of Java's standard library, the interface changes between different Java versions, so extending it directly can become a compatibility issue. Instead, we can implement a KafkaChannel interface, which will implement connect(), read(), write() and possibly other methods we use. We will replace direct use of SocketChannel in our code with use of KafkaChannel. Different implementations of KafkaChannel will be instantiated based on the port/SecurityProtocol configuration. This patch will provide at least the PLAINTEXT implementation for KafkaChannel. I will validate that the SSL implementation in KAFKA-1684 can be refactored to use a KafkaChannel interface rather than extend SocketChannel directly. However, the patch will not include the SSL channel itself. The interface should also include setters/getters for principal and remote IP, which will be used for the authentication code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31366: Patch for KAFKA-1461
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31366/#review74866 --- core/src/main/scala/kafka/server/AbstractFetcherThread.scala https://reviews.apache.org/r/31366/#comment121680 OffsetAndDelay / OffsetAndState is a bit confusing, maybe we can just use PartitionFetchState? core/src/main/scala/kafka/server/AbstractFetcherThread.scala https://reviews.apache.org/r/31366/#comment121689 It seems we do not need to pass in the OffsetAndDelay object here as we will create new one anyways. We can still pass in Long, and with that OffsetAndDelay is just internal to AbstractFetcherThread. core/src/main/scala/kafka/server/OffsetAndDelay.scala https://reviews.apache.org/r/31366/#comment121685 Maybe we can just put this case class into AbstractFetcherThread and expose to AbstractFetcherManager. core/src/main/scala/kafka/server/ReplicaFetcherThread.scala https://reviews.apache.org/r/31366/#comment121686 Are these imports necessary? core/src/main/scala/kafka/server/ReplicaFetcherThread.scala https://reviews.apache.org/r/31366/#comment121688 Is this intentional? - Guozhang Wang On Feb. 24, 2015, 6:02 p.m., Sriharsha Chintalapani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31366/ --- (Updated Feb. 24, 2015, 6:02 p.m.) Review request for kafka. Bugs: KAFKA-1461 https://issues.apache.org/jira/browse/KAFKA-1461 Repository: kafka Description --- KAFKA-1461. Replica fetcher thread does not implement any back-off behavior. Diffs - core/src/main/scala/kafka/server/AbstractFetcherManager.scala 20c00cb8cc2351950edbc8cb1752905a0c26e79f core/src/main/scala/kafka/server/AbstractFetcherThread.scala 8c281d4668f92eff95a4a5df3c03c4b5b20e7095 core/src/main/scala/kafka/server/KafkaConfig.scala 14bf3216bae030331bdf76b3266ed0e73526c3de core/src/main/scala/kafka/server/OffsetAndDelay.scala PRE-CREATION core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 6879e730282185bda3d6bc3659cb15af0672cecf core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala da4bafc1e2a94a436efe395aab1888fc21e55748 Diff: https://reviews.apache.org/r/31366/diff/ Testing --- Thanks, Sriharsha Chintalapani
Re: Review Request 29467: Patch for KAFKA-1660
On March 2, 2015, 11:04 p.m., Jiangjie Qin wrote: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java, line 219 https://reviews.apache.org/r/29467/diff/4/?file=882250#file882250line219 We probably need to release the caller threads that are waiting on flush() at this point. Making flush a boolean method that you have to always check to see if someone called close() in another thead would be a really really really painful api to use in practice, right? I think the issue here is actually what I pointed out in the other comment, namely that in-flight requests area actually left incomplete when you call close and hit the forceClose timeout. Any other thread blocking on these futures would block forever. The right solution is just to fail all requests that haven't completed when forceClose kicks in. This then fullfills the criteria for flush which is that all the requests are completed or failed. - Jay --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74843 --- On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
Re: Review Request 29467: Patch for KAFKA-1660
On March 2, 2015, 11:04 p.m., Jiangjie Qin wrote: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java, line 219 https://reviews.apache.org/r/29467/diff/4/?file=882250#file882250line219 We probably need to release the caller threads that are waiting on flush() at this point. Jay Kreps wrote: Making flush a boolean method that you have to always check to see if someone called close() in another thead would be a really really really painful api to use in practice, right? I think the issue here is actually what I pointed out in the other comment, namely that in-flight requests area actually left incomplete when you call close and hit the forceClose timeout. Any other thread blocking on these futures would block forever. The right solution is just to fail all requests that haven't completed when forceClose kicks in. This then fullfills the criteria for flush which is that all the requests are completed or failed. Yes, I agree that letting flush() return a boolean to just indicate whether someone called close is ugly. I'm thinking maybe we can make the return value to be more useful. The idea of letting flush return a boolean comes when I was writing the mirror maker. When we call flush() followed by a consumer.commitOffsets(), we need to know the result of flush() in order to decide whether to commit offset or not. There might be three cases: 1. flush() succeeded on all batches. 2. flush() failed and some exception were thrown to caller thread (very rare, InterruptedException maybe) 3. flush() failed but are handled by sender thread in send callbacks. For 1), no problem, everybody is happy. For 2), caller thread knows something wrong happened and will not do next task (i.e. commit offsets). For 3), caller thread has no idea about what happened and assumes everthing went well. What I'm doing now is in send callback let the sender thread set a flag for the caller thread to check whether the flush succeeded or not when flush() returns. Otherwise, caller thread cannot decide whether to commit offset or not. I'm thinking if in most cases people care about whether flush succeeded or not, they need to have this inter thread communication. If it is a common requirement, maybe we can let flush() return a boolean. From API point of view, it is probably OK. If user cares about whether flush succeeded or not, they check the return value, otherwise they ignore it. Just like the what we do for send(). - Jiangjie --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74843 --- On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.
[
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344481#comment-14344481
]
Jay Kreps commented on KAFKA-1660:
--
[~parth.brahmbhatt] Yeah this was exactly what I was thinking. It would be good
to add some tests for it and kick off the KIP discussion.
[~guozhang] It looks to me like this should work if called from within a
Callback, but I think you guys would have to specifically try that case or add
a unit test for it. It would be good if you guys can do a pass on the code
review once there are some tests.
Ability to call close() with a timeout on the Java Kafka Producer.
---
Key: KAFKA-1660
URL: https://issues.apache.org/jira/browse/KAFKA-1660
Project: Kafka
Issue Type: Improvement
Components: clients, producer
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Parth Brahmbhatt
Fix For: 0.8.3
Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch,
KAFKA-1660_2015-03-02_10:41:49.patch
I would like the ability to call {{close}} with a timeout on the Java
Client's KafkaProducer.
h6. Workaround
Currently, it is possible to ensure that {{close}} will return quickly by
first doing a {{future.get(timeout)}} on the last future produced on each
partition, but this means that the user has to define the partitions up front
at the time of {{send}} and track the returned {{future}}'s
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1973) Remove the accidentally created LogCleanerManager.scala.orig
[ https://issues.apache.org/jira/browse/KAFKA-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KAFKA-1973: --- Attachment: KAFKA-1973.patch Remove the accidentally created LogCleanerManager.scala.orig Key: KAFKA-1973 URL: https://issues.apache.org/jira/browse/KAFKA-1973 Project: Kafka Issue Type: Bug Reporter: Jiangjie Qin Attachments: KAFKA-1973.patch It seems there is a LogCleanerManager.scala.orig in the trunk now. Need to remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1973) Remove the accidentally created LogCleanerManager.scala.orig
[ https://issues.apache.org/jira/browse/KAFKA-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KAFKA-1973: --- Status: Patch Available (was: Open) Removes core/src/main/scala/kafka/log/LogCleanerManager.scala.orig Remove the accidentally created LogCleanerManager.scala.orig Key: KAFKA-1973 URL: https://issues.apache.org/jira/browse/KAFKA-1973 Project: Kafka Issue Type: Bug Reporter: Jiangjie Qin Attachments: KAFKA-1973.patch It seems there is a LogCleanerManager.scala.orig in the trunk now. Need to remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29467: Patch for KAFKA-1660
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74884 --- clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java https://reviews.apache.org/r/29467/#comment121709 It's probably worth adding an if(timeout 0) on this. clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java https://reviews.apache.org/r/29467/#comment121708 This seems to call initiateClose() twice, once in initiateClose and then again from forceClose. This seems like it depends on all the things getting closed being idempotent to repeated calls (e.g. record accumulator etc). Would it make more sense to have forceClose() just set the force flag? Two minor changes I noted, but otherwise looks good to me. Needs some unit tests, as you mentioned. - Jay Kreps On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
Re: Review Request 29467: Patch for KAFKA-1660
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74897
---
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
https://reviews.apache.org/r/29467/#comment121725
Now there is a bit of duplicate code between the two close methods. Maybe
this would be cleaner if we just made
public void close() {
close(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
}
- Jay Kreps
On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/
---
(Updated March 2, 2015, 6:41 p.m.)
Review request for kafka.
Bugs: KAFKA-1660
https://issues.apache.org/jira/browse/KAFKA-1660
Repository: kafka
Description
---
Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
Conflicts:
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
Changing log levels as suggested.
Diffs
-
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
7397e565fd865214529ffccadd4222d835ac8110
clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java
6913090af03a455452b0b5c3df78f266126b3854
clients/src/main/java/org/apache/kafka/clients/producer/Producer.java
5b3e75ed83a940bc922f9eca10d4008d67aa37c9
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
ed9c63a6679e3aaf83d19fde19268553a4c107c2
Diff: https://reviews.apache.org/r/29467/diff/
Testing
---
existing unit tests passed.
Thanks,
Parth Brahmbhatt
Re: Review Request 31369: Patch for KAFKA-1982
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/#review74895 --- clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java https://reviews.apache.org/r/31369/#comment121727 Could we add a unit test for Integer Ser/DeSer? clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java https://reviews.apache.org/r/31369/#comment121724 Incorrect comment. clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java https://reviews.apache.org/r/31369/#comment121722 Incorrect comment. examples/src/main/java/kafka/examples/Producer.java https://reviews.apache.org/r/31369/#comment121726 We should handle the case when metadata is null. - Jun Rao On Feb. 27, 2015, 7:08 p.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/ --- (Updated Feb. 27, 2015, 7:08 p.m.) Review request for kafka. Bugs: KAFKA-1982 https://issues.apache.org/jira/browse/KAFKA-1982 Repository: kafka Description --- KAFKA-1982: change kafka.examples.Producer to use the new java producer Diffs - clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java PRE-CREATION examples/src/main/java/kafka/examples/Consumer.java 13135b954f3078eeb7394822b0db25470b746f03 examples/src/main/java/kafka/examples/Producer.java 96e98933148d07564c1b30ba8e805e2433c2adc8 Diff: https://reviews.apache.org/r/31369/diff/ Testing --- Thanks, Ashish Singh
[jira] [Resolved] (KAFKA-952) a broker should unregister certain ZK watchers afte it is no longer the controller
[ https://issues.apache.org/jira/browse/KAFKA-952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-952. --- Resolution: Duplicate This is already fixed in KAFKA-1578. a broker should unregister certain ZK watchers afte it is no longer the controller -- Key: KAFKA-952 URL: https://issues.apache.org/jira/browse/KAFKA-952 Project: Kafka Issue Type: Bug Components: controller Affects Versions: 0.8.1 Reporter: Jun Rao Assignee: Geoffrey Anderson Labels: newbie It seems that we only register watchers in the controller logic, but never deregister any watchers. Technically, after a broker stops becoming a controller, the only watcher that it needs to keep registering is on the controller path. The rest of the watchers can be deregistered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31369: Patch for KAFKA-1982
On Feb. 27, 2015, 7:29 p.m., Gwen Shapira wrote: Thanks for the patch, Ashish. Its shaping up to be a very useful example. Two comments: 1. I think the ser/de should be part of the example and not in common, I'm not sure integer ser/de is useful enough to be distributed with Kafka (although Jun can correct me if I got this wrong). 2. I saw a lot of discussion on the mailing list around using the new producer async vs. sync. This example shows the async path. Do we want to add another sync example where we do something like: val future = producer.send(new ProducerRecordInteger, String(topic, messageNo, messageStr), new DemoCallBack(startTime, messageNo, messageStr)); // this waits for send to complete future.get Gwen, Integer may be a common type for keys. So, it probably makes sense to include Integer ser/de in common. I agree that it would be useful to add a sync example. - Jun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/#review74553 --- On Feb. 27, 2015, 7:08 p.m., Ashish Singh wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31369/ --- (Updated Feb. 27, 2015, 7:08 p.m.) Review request for kafka. Bugs: KAFKA-1982 https://issues.apache.org/jira/browse/KAFKA-1982 Repository: kafka Description --- KAFKA-1982: change kafka.examples.Producer to use the new java producer Diffs - clients/src/main/java/org/apache/kafka/common/serialization/IntegerDeserializer.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/serialization/IntegerSerializer.java PRE-CREATION examples/src/main/java/kafka/examples/Consumer.java 13135b954f3078eeb7394822b0db25470b746f03 examples/src/main/java/kafka/examples/Producer.java 96e98933148d07564c1b30ba8e805e2433c2adc8 Diff: https://reviews.apache.org/r/31369/diff/ Testing --- Thanks, Ashish Singh
Re: Review Request 31566: Patch for KAFKA-1988
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/#review74889 --- Ship it! This looks good assuming the other patch, centralizes the scala code to all use this single abs function. - Jay Kreps On Feb. 27, 2015, 11:16 p.m., Tong Li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ --- (Updated Feb. 27, 2015, 11:16 p.m.) Review request for kafka. Bugs: KAFKA-1988 https://issues.apache.org/jira/browse/KAFKA-1988 Repository: kafka Description --- KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 69530c187cd1c41b8173b61de6f982aafe65c9fe clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f Diff: https://reviews.apache.org/r/31566/diff/ Testing --- Thanks, Tong Li
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344500#comment-14344500 ] Jay Kreps commented on KAFKA-902: - This looks good to me. I'd second Grant's comments: 1. I agree we should probably make it configurable and mark the configuration low importance. This kind of configuration is hyper-annoying because no one will ever set it but it's probably the right thing to do. 2. We should definitely apply the same thing to the reconnect backoff as well as metadata max age (if everyone disconnects at time X they will all expire their metadata at X+metadata.max.age.ms so jittering that will help too). Another thing is that this jitter is only additive, so if you configure a backoff of 10 ms, your observed backoff time will be 15 ms. Also 10 ms will be a bit large if you configure a 1 ms backoff and zero ends up being kind of magical. I don't think this is really too terrible and it is simple, so maybe we should just leave it. Another possibility would be something like using a jitter that is a random int in +/- min(20, 0.2 * backoff_ms). Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31566: Patch for KAFKA-1988
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/#review74893 --- Thanks for the patch. A couple of minor comments below. clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java https://reviews.apache.org/r/31566/#comment121720 Perhaps we can change the comment to the following. A cheap way to deterministically convert a number to a positive value. When the input number is negative, the returned positive value is not the absolute value of the input though. clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java https://reviews.apache.org/r/31566/#comment121721 We can just say it returns a positive number. - Jun Rao On Feb. 27, 2015, 11:16 p.m., Tong Li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31566/ --- (Updated Feb. 27, 2015, 11:16 p.m.) Review request for kafka. Bugs: KAFKA-1988 https://issues.apache.org/jira/browse/KAFKA-1988 Repository: kafka Description --- KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 69530c187cd1c41b8173b61de6f982aafe65c9fe clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f Diff: https://reviews.apache.org/r/31566/diff/ Testing --- Thanks, Tong Li
[jira] [Commented] (KAFKA-1994) Evaluate performance effect of chroot check on Topic creation
[ https://issues.apache.org/jira/browse/KAFKA-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344572#comment-14344572 ] Jun Rao commented on KAFKA-1994: Ashish, The code path for creating a topic may not be optimized. Could you just test the cost of createPersistent() with and w/o the patch? Thanks, Evaluate performance effect of chroot check on Topic creation - Key: KAFKA-1994 URL: https://issues.apache.org/jira/browse/KAFKA-1994 Project: Kafka Issue Type: Improvement Reporter: Ashish K Singh Assignee: Ashish K Singh KAFKA-1664 adds check for chroot while creating a node in ZK. ZkPath checks if namespace exists before trying to create a path in ZK. This raises a concern that checking namespace for each path creation might be unnecessary and can potentially make creations expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29467: Patch for KAFKA-1660
On March 3, 2015, 4:10 a.m., Parth Brahmbhatt wrote: Two minor changes I noted, but otherwise looks good to me. Needs some unit tests, as you mentioned. Actually one probably I didn't think of is that forceClose() leaves the in-flight requests forever incomplete. A better approach would be to fail them all with TimeoutException. - Jay --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74884 --- On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
[jira] [Updated] (KAFKA-1996) Scaladoc error: unknown tag parameter
[ https://issues.apache.org/jira/browse/KAFKA-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yaguo Zhou updated KAFKA-1996: -- Attachment: scala-doc-unknown-tag-parameter.patch Scaladoc error: unknown tag parameter - Key: KAFKA-1996 URL: https://issues.apache.org/jira/browse/KAFKA-1996 Project: Kafka Issue Type: Improvement Components: core Reporter: Yaguo Zhou Priority: Minor Labels: doc Attachments: scala-doc-unknown-tag-parameter.patch There are some scala doc error: unknown tag parameter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations
Hey, I just sent out a google hangout invite to all pmc, committers and everyone I found working on a KIP. If I missed anyone in the invite please let me know and can update it, np. We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA help to make a google account so we can manage better? To discuss https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals in progress and related JIRA that are interdependent and common work. ~ Joe Stein On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps jay.kr...@gmail.com wrote: Let's stay on Google hangouts that will also record and make the sessions available on youtube. -Jay On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman jholo...@cloudera.com wrote: Jay / Joe We're happy to send out a Webex for this purpose. We could record the sessions if there is interest and publish them out. Thanks Jeff On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps jay.kr...@gmail.com wrote: Let's try to get the technical hang-ups sorted out, though. I really think there is some benefit to live discussion vs writing. I am hopeful that if we post instructions and give ourselves a few attempts we can get it working. Tuesday at that time would work for me...any objections? -Jay On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein joe.st...@stealth.ly wrote: Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT I don't mind google hangout but there is always some issue or whatever so we know the apache irc channel works. We can start there and see how it goes? We can pull transcripts too and associate to tickets if need be makes it helpful for things. ~ Joestein On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps jay.kr...@gmail.com wrote: We'd talked about doing a Google Hangout to chat about this. What about generalizing that a little further...I actually think it would be good for everyone spending a reasonable chunk of their week on Kafka stuff to maybe sync up once a week. I think we could use time to talk through design stuff, make sure we are on top of code reviews, talk through any tricky issues, etc. We can make it publicly available so that any one can follow along who likes. Any interest in doing this? If so I'll try to set it up starting next week. -Jay On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Hi all, I've updated KIP page, fixed / aligned document structure. Also I added some very initial proposal for AdminClient so we have something to start from while discussing the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations Thanks, Andrii Biletskyi On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Jay, Re error messages: you are right, in most cases client will have enough context to show descriptive error message. My concern is that we will have to add lots of new error codes for each possible error. Of course, we could reuse some of existing like UknownTopicOrPartitionCode, but we will also need to add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both for topic name and config, and probably user would like to know what exactly is wrong in his config), InvalidReplicaAssignment, InternalError (e.g. zookeeper failure) etc. And this is only for TopicCommand, we will also need to add similar stuff for ReassignPartitions, PreferredReplica. So we'll end up with a large list of error codes, used only in Admin protocol. Having said that, I agree my proposal is not consistent with other cases. Maybe we can find better solution or something in-between. Re Hangout chat: I think it is a great idea. This way we can move on faster. Let's agree somehow on date/time so people can join. Will work for me this and next week almost anytime if agreed in advance. Thanks, Andrii On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Andrii, Generally we can do good error handling without needing custom server-side messages. I.e. generally the client has the context to know that if it got an error that the topic doesn't exist to say Topic X doesn't exist rather than error code 14 (or whatever). Maybe there are specific cases where this is hard? If we want to add server-side
[jira] [Created] (KAFKA-1996) Scaladoc error: unknown tag parameter
Yaguo Zhou created KAFKA-1996: - Summary: Scaladoc error: unknown tag parameter Key: KAFKA-1996 URL: https://issues.apache.org/jira/browse/KAFKA-1996 Project: Kafka Issue Type: Improvement Components: core Reporter: Yaguo Zhou Priority: Minor There are some scala doc error: unknown tag parameter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343109#comment-14343109 ] Tong Li commented on KAFKA-1988: [~guozhang] Yes, the o.a.k.c.c.u.Utils.abs used in few places. in patch set for issue 1926, I will consolidate both Utils modules from clients and core into one. So that we do not have name conflict all over the place. The patch set for issue 1926 will be quite big. I would like to get this thing fixed for coming up release first, then we can address issue 1926. Thanks. org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1884) New Producer blocks forever for Invalid topic names
[
https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar Reddy updated KAFKA-1884:
---
Assignee: Manikumar Reddy
Status: Patch Available (was: Open)
New Producer blocks forever for Invalid topic names
---
Key: KAFKA-1884
URL: https://issues.apache.org/jira/browse/KAFKA-1884
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8.2.0
Reporter: Manikumar Reddy
Assignee: Manikumar Reddy
Fix For: 0.8.3
Attachments: KAFKA-1884.patch
New producer blocks forever for invalid topics names
producer logs:
DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50845,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,416] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50845.
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50846,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,417] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50846.
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,418] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50847,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,418] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50847.
Broker logs:
[2015-01-20 12:46:14,074] ERROR [KafkaApi-0] error when handling request
Name: TopicMetadataRequest; Version: 0; CorrelationId: 51020; ClientId:
my-producer; Topics: TOPIC= (kafka.server.KafkaApis)
kafka.common.InvalidTopicException: topic name TOPIC= is illegal, contains a
character other than ASCII alphanumerics, '.', '_' and '-'
at kafka.common.Topic$.validate(Topic.scala:42)
at
kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:186)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:177)
at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:367)
at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:350)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:350)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:389)
at kafka.server.KafkaApis.handle(KafkaApis.scala:57)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:722)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1884) New Producer blocks forever for Invalid topic names
[
https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343294#comment-14343294
]
Manikumar Reddy commented on KAFKA-1884:
Created reviewboard https://reviews.apache.org/r/31627/diff/
against branch origin/trunk
New Producer blocks forever for Invalid topic names
---
Key: KAFKA-1884
URL: https://issues.apache.org/jira/browse/KAFKA-1884
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8.2.0
Reporter: Manikumar Reddy
Fix For: 0.8.3
Attachments: KAFKA-1884.patch
New producer blocks forever for invalid topics names
producer logs:
DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50845,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,416] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50845.
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50846,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,417] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50846.
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,418] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50847,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,418] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50847.
Broker logs:
[2015-01-20 12:46:14,074] ERROR [KafkaApi-0] error when handling request
Name: TopicMetadataRequest; Version: 0; CorrelationId: 51020; ClientId:
my-producer; Topics: TOPIC= (kafka.server.KafkaApis)
kafka.common.InvalidTopicException: topic name TOPIC= is illegal, contains a
character other than ASCII alphanumerics, '.', '_' and '-'
at kafka.common.Topic$.validate(Topic.scala:42)
at
kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:186)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:177)
at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:367)
at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:350)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:350)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:389)
at kafka.server.KafkaApis.handle(KafkaApis.scala:57)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:722)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1877) Expose version via JMX for 'new' producer
[
https://issues.apache.org/jira/browse/KAFKA-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343275#comment-14343275
]
Manikumar Reddy commented on KAFKA-1877:
Yes, version info can be exposed as JMX info. If some one want to
programmatically retrieve the version info, how to retrieve?
Expose version via JMX for 'new' producer
--
Key: KAFKA-1877
URL: https://issues.apache.org/jira/browse/KAFKA-1877
Project: Kafka
Issue Type: Bug
Components: clients, producer
Affects Versions: 0.8.2.0
Reporter: Vladimir Tretyakov
Assignee: Manikumar Reddy
Fix For: 0.8.3
Add version of Kafka to jmx (monitoring tool can use this info).
Something like that
{code}
kafka.common:type=AppInfo,name=Version
Value java.lang.Object = 0.8.2-beta
{code}
we already have this in core Kafka module (see kafka.common.AppInfo object).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1884) New Producer blocks forever for Invalid topic names
[
https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar Reddy updated KAFKA-1884:
---
Attachment: KAFKA-1884.patch
New Producer blocks forever for Invalid topic names
---
Key: KAFKA-1884
URL: https://issues.apache.org/jira/browse/KAFKA-1884
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8.2.0
Reporter: Manikumar Reddy
Fix For: 0.8.3
Attachments: KAFKA-1884.patch
New producer blocks forever for invalid topics names
producer logs:
DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,406] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50845,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,416] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50845.
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50846,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,417] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50846.
DEBUG [2015-01-20 12:46:13,417] NetworkClient: maybeUpdateMetadata(): Trying
to send metadata request to node -1
DEBUG [2015-01-20 12:46:13,418] NetworkClient: maybeUpdateMetadata(): Sending
metadata request ClientRequest(expectResponse=true, payload=null,
request=RequestSend(header={api_key=3,api_version=0,correlation_id=50847,client_id=my-producer},
body={topics=[TOPIC=]})) to node -1
TRACE [2015-01-20 12:46:13,418] NetworkClient: handleMetadataResponse():
Ignoring empty metadata response with correlation id 50847.
Broker logs:
[2015-01-20 12:46:14,074] ERROR [KafkaApi-0] error when handling request
Name: TopicMetadataRequest; Version: 0; CorrelationId: 51020; ClientId:
my-producer; Topics: TOPIC= (kafka.server.KafkaApis)
kafka.common.InvalidTopicException: topic name TOPIC= is illegal, contains a
character other than ASCII alphanumerics, '.', '_' and '-'
at kafka.common.Topic$.validate(Topic.scala:42)
at
kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:186)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:177)
at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:367)
at kafka.server.KafkaApis$$anonfun$5.apply(KafkaApis.scala:350)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at
scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:350)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:389)
at kafka.server.KafkaApis.handle(KafkaApis.scala:57)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:722)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 28481: Patch for KAFKA-1792
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28481/#review74760 --- core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala https://reviews.apache.org/r/28481/#comment121477 I'm not sure if this makes sense. Basically, the entire point of --rebalance is to figure out the best balanced replica placement with the minimum number of replicas moved. If you ask the user to list the topics or the brokers, this may not lead to the most balanced replica placement in the cluster. If we did this, then the only thing the user would want to do is limit the number of replicas moved in one go, in order to manually throttle the data movement in the cluster. It is ok to do that in a separate JIRA. Same with the replace broker use case. Replacing a broker is much easier to use if it is a separate option (--replace-broker --from-broker 1 --to-broker 2). Though if you want to cover that in a separate JIRA, that's fine. - Neha Narkhede On Feb. 26, 2015, 2:58 p.m., Dmitry Pekar wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28481/ --- (Updated Feb. 26, 2015, 2:58 p.m.) Review request for kafka. Bugs: KAFKA-1792 https://issues.apache.org/jira/browse/KAFKA-1792 Repository: kafka Description --- KAFKA-1792: CR KAFKA-1792: CR2 KAFKA-1792: merge of KAFKA-1753 KAFKA-1792: generate renamed to rebalance KAFKA-1792: --rebalance renamed back to --generate, removed --decomission-broker command KAFKA-1792: added back --decommission-broker command KAFKA-1792: --generate renamed back to --rebalance KAFKA-1792: added old --generate command for compatibility Diffs - core/src/main/scala/kafka/admin/AdminUtils.scala b700110f2d7f1ede235af55d8e37e1b5592c6c7d core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 979992b68af3723cd229845faff81c641123bb88 core/src/test/scala/unit/kafka/admin/AdminTest.scala e28979827110dfbbb92fe5b152e7f1cc973de400 topics.json ff011ed381e781b9a177036001d44dca3eac586f Diff: https://reviews.apache.org/r/28481/diff/ Testing --- Thanks, Dmitry Pekar
Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations
Thanks for sending this out Joe. Looking forward to chatting with everyone :) On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein joe.st...@stealth.ly wrote: Hey, I just sent out a google hangout invite to all pmc, committers and everyone I found working on a KIP. If I missed anyone in the invite please let me know and can update it, np. We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA help to make a google account so we can manage better? To discuss https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals in progress and related JIRA that are interdependent and common work. ~ Joe Stein On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps jay.kr...@gmail.com wrote: Let's stay on Google hangouts that will also record and make the sessions available on youtube. -Jay On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman jholo...@cloudera.com wrote: Jay / Joe We're happy to send out a Webex for this purpose. We could record the sessions if there is interest and publish them out. Thanks Jeff On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps jay.kr...@gmail.com wrote: Let's try to get the technical hang-ups sorted out, though. I really think there is some benefit to live discussion vs writing. I am hopeful that if we post instructions and give ourselves a few attempts we can get it working. Tuesday at that time would work for me...any objections? -Jay On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein joe.st...@stealth.ly wrote: Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT I don't mind google hangout but there is always some issue or whatever so we know the apache irc channel works. We can start there and see how it goes? We can pull transcripts too and associate to tickets if need be makes it helpful for things. ~ Joestein On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps jay.kr...@gmail.com wrote: We'd talked about doing a Google Hangout to chat about this. What about generalizing that a little further...I actually think it would be good for everyone spending a reasonable chunk of their week on Kafka stuff to maybe sync up once a week. I think we could use time to talk through design stuff, make sure we are on top of code reviews, talk through any tricky issues, etc. We can make it publicly available so that any one can follow along who likes. Any interest in doing this? If so I'll try to set it up starting next week. -Jay On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Hi all, I've updated KIP page, fixed / aligned document structure. Also I added some very initial proposal for AdminClient so we have something to start from while discussing the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations Thanks, Andrii Biletskyi On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Jay, Re error messages: you are right, in most cases client will have enough context to show descriptive error message. My concern is that we will have to add lots of new error codes for each possible error. Of course, we could reuse some of existing like UknownTopicOrPartitionCode, but we will also need to add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both for topic name and config, and probably user would like to know what exactly is wrong in his config), InvalidReplicaAssignment, InternalError (e.g. zookeeper failure) etc. And this is only for TopicCommand, we will also need to add similar stuff for ReassignPartitions, PreferredReplica. So we'll end up with a large list of error codes, used only in Admin protocol. Having said that, I agree my proposal is not consistent with other cases. Maybe we can find better solution or something in-between. Re Hangout chat: I think it is a great idea. This way we can move on faster. Let's agree somehow on date/time so people can join. Will work for me this and next week almost anytime if agreed in advance. Thanks, Andrii On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey Andrii, Generally we can do good error handling without needing custom server-side messages. I.e. generally the client has the context to know that if it got an error that the topic doesn't exist to say Topic X doesn't
[jira] [Created] (KAFKA-1997) Refactor Mirror Maker
Jiangjie Qin created KAFKA-1997: --- Summary: Refactor Mirror Maker Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29467: Patch for KAFKA-1660
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/#review74777 --- clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java https://reviews.apache.org/r/29467/#comment121523 Changed log level as suggested. clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java https://reviews.apache.org/r/29467/#comment121524 included. clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java https://reviews.apache.org/r/29467/#comment121525 changed log level to suggested value. - Parth Brahmbhatt On March 2, 2015, 6:41 p.m., Parth Brahmbhatt wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
Apache Samza Meetup - March 4 @6PM hosted at LinkedIn's campus in Mountain View CA
Hi all - I would like to announce the first Bay Area Apache Samza Meetuphttp://www.meetup.com/Bay-Area-Samza-Meetup/events/220354853/ hosted at LinkedIn in Mountain View, CA on March 4, 2015 @6PM. We plan to host the event every 2-months to encourage knowledge sharing collaboration in Samza’s usagehttp://wiki.apache.org/samza/PoweredBy and open sourcehttp://samza.apache.org/ community.http://samza.apache.org/ The agenda for the meetup is:: * 6:00 – 6:15PM: Doors open, sign NDAs, networking, food drinks * 6:15 - 6:45PM: Naveen Somasundaram (LinkedIn) – Getting Started with Apache Samza * 6:45 - 7:30PM: Shekar Tippur (Intuit) – Powering Contextual Alerts for Intuit’s Operations Center with Apache Samza * 7:30 - 8:00PM: Chris Riccomini (LinkedIn) – Where we’re headed next: Apache Samza Roadmap We plan to provide food drinks so please RSVP herehttp://www.meetup.com/Bay-Area-Samza-Meetup/events/220354853/ to help us with estimation. Please let me know if you have any questions or ideas for future meet ups. We plan to announce a live stream the day of the event for remote attendance. Excited to see you there! Ed Yakabosky [BCC: Kafka Open Source Samza Open Source LinkedIn’s DDS and DAI teams Linkedin’s Samza customers Tech-Talk]
[jira] [Commented] (KAFKA-1997) Refactor Mirror Maker
[ https://issues.apache.org/jira/browse/KAFKA-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343488#comment-14343488 ] Jonathan Creasy commented on KAFKA-1997: https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement Refactor Mirror Maker - Key: KAFKA-1997 URL: https://issues.apache.org/jira/browse/KAFKA-1997 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin Assignee: Jiangjie Qin Refactor mirror maker based on KIP-3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.
[
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Parth Brahmbhatt updated KAFKA-1660:
Attachment: KAFKA-1660_2015-03-02_10:41:49.patch
Ability to call close() with a timeout on the Java Kafka Producer.
---
Key: KAFKA-1660
URL: https://issues.apache.org/jira/browse/KAFKA-1660
Project: Kafka
Issue Type: Improvement
Components: clients, producer
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Parth Brahmbhatt
Fix For: 0.8.3
Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch,
KAFKA-1660_2015-03-02_10:41:49.patch
I would like the ability to call {{close}} with a timeout on the Java
Client's KafkaProducer.
h6. Workaround
Currently, it is possible to ensure that {{close}} will return quickly by
first doing a {{future.get(timeout)}} on the last future produced on each
partition, but this means that the user has to define the partitions up front
at the time of {{send}} and track the returned {{future}}'s
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29467: Patch for KAFKA-1660
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29467/ --- (Updated March 2, 2015, 6:41 p.m.) Review request for kafka. Bugs: KAFKA-1660 https://issues.apache.org/jira/browse/KAFKA-1660 Repository: kafka Description (updated) --- Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Conflicts: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java Merge remote-tracking branch 'origin/trunk' into KAFKA-1660 Changing log levels as suggested. Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 6913090af03a455452b0b5c3df78f266126b3854 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5b3e75ed83a940bc922f9eca10d4008d67aa37c9 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ed9c63a6679e3aaf83d19fde19268553a4c107c2 Diff: https://reviews.apache.org/r/29467/diff/ Testing --- existing unit tests passed. Thanks, Parth Brahmbhatt
[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.
[
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343552#comment-14343552
]
Parth Brahmbhatt commented on KAFKA-1660:
-
Updated reviewboard https://reviews.apache.org/r/29467/diff/
against branch origin/trunk
Ability to call close() with a timeout on the Java Kafka Producer.
---
Key: KAFKA-1660
URL: https://issues.apache.org/jira/browse/KAFKA-1660
Project: Kafka
Issue Type: Improvement
Components: clients, producer
Affects Versions: 0.8.2.0
Reporter: Andrew Stein
Assignee: Parth Brahmbhatt
Fix For: 0.8.3
Attachments: KAFKA-1660.patch, KAFKA-1660_2015-02-17_16:41:19.patch,
KAFKA-1660_2015-03-02_10:41:49.patch
I would like the ability to call {{close}} with a timeout on the Java
Client's KafkaProducer.
h6. Workaround
Currently, it is possible to ensure that {{close}} will return quickly by
first doing a {{future.get(timeout)}} on the last future produced on each
partition, but this means that the user has to define the partitions up front
at the time of {{send}} and track the returned {{future}}'s
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343607#comment-14343607 ] Geoffrey Anderson commented on KAFKA-902: - Created reviewboard https://reviews.apache.org/r/31633/diff/ against branch origin/trunk Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Status: Patch Available (was: Open) Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31633: Patch for KAFKA-902
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31633/ --- Review request for kafka. Bugs: KAFKA-902 https://issues.apache.org/jira/browse/KAFKA-902 Repository: kafka Description --- Add simple unit test for ClientUtils.randomizeBackoff Diffs - clients/src/main/java/org/apache/kafka/clients/ClientUtils.java d0da5d7a08a0c3e67e0fe14bb0b0e7c73380f416 clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 67ceb754a52c07143c69b053fe128b9e24060b99 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 7397e565fd865214529ffccadd4222d835ac8110 clients/src/test/java/org/apache/kafka/clients/ClientUtilsTest.java 13ce519f03d13db041e1f2dbcd6b59414d2775b8 Diff: https://reviews.apache.org/r/31633/diff/ Testing --- Thanks, Geoffrey Anderson
[jira] [Updated] (KAFKA-902) Randomize backoff on the clients for metadata requests
[ https://issues.apache.org/jira/browse/KAFKA-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Anderson updated KAFKA-902: Attachment: KAFKA-902.patch Randomize backoff on the clients for metadata requests -- Key: KAFKA-902 URL: https://issues.apache.org/jira/browse/KAFKA-902 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.0 Reporter: Neha Narkhede Assignee: Geoffrey Anderson Priority: Critical Labels: newbie Attachments: KAFKA-902.patch If a Kafka broker dies and there are a large number of clients talking to the Kafka cluster, each of the clients can end up shooting metadata requests at around the same time. It is better to randomize the backoff on the clients so the metadata requests are more evenly spread out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
