Re: Review Request 30763: KAFKA-1865: Producer Flush: Remove synchronization.

[Ewen Cheslack-Postava]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30763/#review74229
---



clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java


synchronized no longer matches the regular non-Mock version.


- Ewen Cheslack-Postava


On Feb. 26, 2015, 1:16 a.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30763/
> ---
> 
> (Updated Feb. 26, 2015, 1:16 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1865
> https://issues.apache.org/jira/browse/KAFKA-1865
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1865 Add a flush() method to the producer.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/Metadata.java 
> e8afecda956303a6ee116499fd443a54c018e17d 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 1fd6917c8a5131254c740abad7f7228a47e3628c 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 84530f2b948f9abd74203db48707e490dd9c81a5 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 17fe541588d462c68c33f6209717cc4015e9b62f 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerRecord.java 
> 4990692efa6f01c62e1d7b05fbf31bec50e398c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/FutureRecordMetadata.java
>  4a2da41f47994f778109e3c4107ffd90195f0bae 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  ecfe2144d778a5d9b614df5278b9f0a15637f10b 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  dd0af8aee98abed5d4a0dc50989e37888bb353fe 
>   
> clients/src/main/java/org/apache/kafka/common/errors/InterruptException.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/utils/SystemTime.java 
> d682bd46ec3826f0a72388cc4ec30e1b1223d0f3 
>   clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java 
> 4ae43ed47e31ad8052b4348a731da11120968508 
>   clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
> 743aa7e523dd476949f484bfa4c7fb8a3afd7bf8 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java 
> 75513b0bdd439329c5771d87436ef83fda853bfb 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 
> 404bedb3dc4e44cc79251d71e1e3f8efdab60efa 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  83338633717cfa4ef7cf2a590b5aa6b9c8cb1dd2 
>   clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
> 558942aaecd1b9f7098435d39aa4b362cd16ff0a 
>   core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
> 2802a399bf599e9530f53b7df72f12702a10d3c4 
>   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
> b15237b76def3b234924280fa3fdb25dbb0cc0dc 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 21d0ed2cb7c9459261d3cdc7c21dece5e2079698 
> 
> Diff: https://reviews.apache.org/r/30763/diff/
> 
> 
> Testing
> ---
> 
> The latest patch uses Jiangjie's suggestion to remove the synchronization on 
> flush.
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

Re: Review Request 31447: Patch for KAFKA-1881

[Ewen Cheslack-Postava]

> On Feb. 26, 2015, 3:07 a.m., Gwen Shapira wrote:
> > Ah, now I get what "dedupe buffer" refers to :)
> > 
> > I never managed to reproduce the oom, but this looks like the right fix.
> 
> Ewen Cheslack-Postava wrote:
> This happened consistently for me, which provided good motivation to 
> figure out how to fix it :) It looks like the tests use the default JVM 
> settings (build.gradle has overrides for compile, but not for test), so the 
> 500 MB allocation was always failing since it obviously exceeds the default 
> 128 MB heap. Now I'm wondering if other people are running with different 
> settings. Maybe if you have a gradle daemon running that was started with a 
> bigger max heap size?
> 
> Gwen Shapira wrote:
> My gradle.properties has:
> org.gradle.jvmargs=-XX:MaxPermSize=512m -Xmx1024m
> 
> I take it that yours doesn't? 
> 
> The builds in Apache's jenkins didn't seem to fail on that test either...
> 
> Ewen Cheslack-Postava wrote:
> The one checked into the repo does, and I double checked that I'm not 
> overriding that setting in my ~/.gradle/gradle.properties. But as far as I 
> can tell that only applies to the gradle daemon process. It forks off another 
> process to run the tests, which doesn't inherit those settings. If I add this 
> to the build.gradle file next to the overrides for ScalaCompile tasks:
> 
> tasks.withType(JavaForkOptions) {
> jvmArgs = ['-XX:MaxPermSize=512m', '-Xmx1024m']
> }
>   
> then the forked process gets those settings and the test passes without 
> this patch.
> 
> Gwen Shapira wrote:
> huh. so I have no idea why the tests pass on my setup...

I checked my setup by grabbing info from visualvm while the tests were running. 
Maybe double check the processes running for you when you run the tests, and 
what settings those processes have had passed to them?

I don't think it's critical since the immediate issue is resolved by the patch, 
but it would be nice to know why our test environments are inconsistent.


- Ewen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31447/#review74200
---


On Feb. 25, 2015, 11:27 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31447/
> ---
> 
> (Updated Feb. 25, 2015, 11:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1881
> https://issues.apache.org/jira/browse/KAFKA-1881
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1881: Fix out of memory error in testDeleteTopicWithCleaner by making 
> the dedupe buffer much smaller.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 0cbd72684a0cc6fe9f1fba2dc834de675b2eb994 
> 
> Diff: https://reviews.apache.org/r/31447/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

Re: Review Request 30763: KAFKA-1865: Producer Flush: Remove synchronization.

[Jiangjie Qin]

> On Feb. 24, 2015, 5:20 p.m., Guozhang Wang wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java,
> >  line 91
> > 
> >
> > I still think it is kind of waste to create duplicate RecordMetadata 
> > here and in done()..
> 
> Jay Kreps wrote:
> I take your point, but I tried the refactor to remove it and it wasn't 
> very simple plus I think it would be quite rare that you would ever actually 
> invoke both paths. Plus optimizing adds another synchronous memory reference 
> while these super quick object allocations may get optimized out anyway...

Actually in done() we don't create any RecordMetadata... It looks we were 
always creating a RecordMeatadata instance even before this patch. So 
FutureRecordMetatdata.valueOrError() will return a new RecordMeataData. The 
change here is that instead of creating the new RecordMetadata instance in 
valueOrError (which only work after done() is called), we create the 
RecordMeataData instance directly.
I think it is a little bit hacky from the code style point of view, i.e. 
duplicates code. But we are not really creating more RecordMeataData objects 
compared with previous solution.


- Jiangjie


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30763/#review73829
---


On Feb. 26, 2015, 1:16 a.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30763/
> ---
> 
> (Updated Feb. 26, 2015, 1:16 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1865
> https://issues.apache.org/jira/browse/KAFKA-1865
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1865 Add a flush() method to the producer.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/Metadata.java 
> e8afecda956303a6ee116499fd443a54c018e17d 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 1fd6917c8a5131254c740abad7f7228a47e3628c 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 84530f2b948f9abd74203db48707e490dd9c81a5 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 17fe541588d462c68c33f6209717cc4015e9b62f 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerRecord.java 
> 4990692efa6f01c62e1d7b05fbf31bec50e398c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/FutureRecordMetadata.java
>  4a2da41f47994f778109e3c4107ffd90195f0bae 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  ecfe2144d778a5d9b614df5278b9f0a15637f10b 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  dd0af8aee98abed5d4a0dc50989e37888bb353fe 
>   
> clients/src/main/java/org/apache/kafka/common/errors/InterruptException.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/utils/SystemTime.java 
> d682bd46ec3826f0a72388cc4ec30e1b1223d0f3 
>   clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java 
> 4ae43ed47e31ad8052b4348a731da11120968508 
>   clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
> 743aa7e523dd476949f484bfa4c7fb8a3afd7bf8 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java 
> 75513b0bdd439329c5771d87436ef83fda853bfb 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 
> 404bedb3dc4e44cc79251d71e1e3f8efdab60efa 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  83338633717cfa4ef7cf2a590b5aa6b9c8cb1dd2 
>   clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
> 558942aaecd1b9f7098435d39aa4b362cd16ff0a 
>   core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
> 2802a399bf599e9530f53b7df72f12702a10d3c4 
>   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
> b15237b76def3b234924280fa3fdb25dbb0cc0dc 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 21d0ed2cb7c9459261d3cdc7c21dece5e2079698 
> 
> Diff: https://reviews.apache.org/r/30763/diff/
> 
> 
> Testing
> ---
> 
> The latest patch uses Jiangjie's suggestion to remove the synchronization on 
> flush.
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

Re: Review Request 30763: KAFKA-1865: Producer Flush: Remove synchronization.

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30763/#review74196
---


Thanks for the patch. A few comments below.


clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


Not sure how we make sure only a single flush at a time. Could you explain 
a bit more?



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java


Could flushesInProgress just be an AtomicBoolean?



clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java


This assertion doesn't seem cover the case that one of the future is done, 
but the other is not.


- Jun Rao


On Feb. 26, 2015, 1:16 a.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30763/
> ---
> 
> (Updated Feb. 26, 2015, 1:16 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1865
> https://issues.apache.org/jira/browse/KAFKA-1865
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1865 Add a flush() method to the producer.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/Metadata.java 
> e8afecda956303a6ee116499fd443a54c018e17d 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 1fd6917c8a5131254c740abad7f7228a47e3628c 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 84530f2b948f9abd74203db48707e490dd9c81a5 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 17fe541588d462c68c33f6209717cc4015e9b62f 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerRecord.java 
> 4990692efa6f01c62e1d7b05fbf31bec50e398c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/FutureRecordMetadata.java
>  4a2da41f47994f778109e3c4107ffd90195f0bae 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  ecfe2144d778a5d9b614df5278b9f0a15637f10b 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  dd0af8aee98abed5d4a0dc50989e37888bb353fe 
>   
> clients/src/main/java/org/apache/kafka/common/errors/InterruptException.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/utils/SystemTime.java 
> d682bd46ec3826f0a72388cc4ec30e1b1223d0f3 
>   clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java 
> 4ae43ed47e31ad8052b4348a731da11120968508 
>   clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
> 743aa7e523dd476949f484bfa4c7fb8a3afd7bf8 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java 
> 75513b0bdd439329c5771d87436ef83fda853bfb 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 
> 404bedb3dc4e44cc79251d71e1e3f8efdab60efa 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  83338633717cfa4ef7cf2a590b5aa6b9c8cb1dd2 
>   clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
> 558942aaecd1b9f7098435d39aa4b362cd16ff0a 
>   core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
> 2802a399bf599e9530f53b7df72f12702a10d3c4 
>   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
> b15237b76def3b234924280fa3fdb25dbb0cc0dc 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 21d0ed2cb7c9459261d3cdc7c21dece5e2079698 
> 
> Diff: https://reviews.apache.org/r/30763/diff/
> 
> 
> Testing
> ---
> 
> The latest patch uses Jiangjie's suggestion to remove the synchronization on 
> flush.
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

Re: Review Request 31447: Patch for KAFKA-1881

[Gwen Shapira]

> On Feb. 26, 2015, 3:07 a.m., Gwen Shapira wrote:
> > Ah, now I get what "dedupe buffer" refers to :)
> > 
> > I never managed to reproduce the oom, but this looks like the right fix.
> 
> Ewen Cheslack-Postava wrote:
> This happened consistently for me, which provided good motivation to 
> figure out how to fix it :) It looks like the tests use the default JVM 
> settings (build.gradle has overrides for compile, but not for test), so the 
> 500 MB allocation was always failing since it obviously exceeds the default 
> 128 MB heap. Now I'm wondering if other people are running with different 
> settings. Maybe if you have a gradle daemon running that was started with a 
> bigger max heap size?
> 
> Gwen Shapira wrote:
> My gradle.properties has:
> org.gradle.jvmargs=-XX:MaxPermSize=512m -Xmx1024m
> 
> I take it that yours doesn't? 
> 
> The builds in Apache's jenkins didn't seem to fail on that test either...
> 
> Ewen Cheslack-Postava wrote:
> The one checked into the repo does, and I double checked that I'm not 
> overriding that setting in my ~/.gradle/gradle.properties. But as far as I 
> can tell that only applies to the gradle daemon process. It forks off another 
> process to run the tests, which doesn't inherit those settings. If I add this 
> to the build.gradle file next to the overrides for ScalaCompile tasks:
> 
> tasks.withType(JavaForkOptions) {
> jvmArgs = ['-XX:MaxPermSize=512m', '-Xmx1024m']
> }
>   
> then the forked process gets those settings and the test passes without 
> this patch.

huh. so I have no idea why the tests pass on my setup...


- Gwen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31447/#review74200
---


On Feb. 25, 2015, 11:27 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31447/
> ---
> 
> (Updated Feb. 25, 2015, 11:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1881
> https://issues.apache.org/jira/browse/KAFKA-1881
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1881: Fix out of memory error in testDeleteTopicWithCleaner by making 
> the dedupe buffer much smaller.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 0cbd72684a0cc6fe9f1fba2dc834de675b2eb994 
> 
> Diff: https://reviews.apache.org/r/31447/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

Re: Review Request 31447: Patch for KAFKA-1881

[Ewen Cheslack-Postava]

> On Feb. 26, 2015, 3:07 a.m., Gwen Shapira wrote:
> > Ah, now I get what "dedupe buffer" refers to :)
> > 
> > I never managed to reproduce the oom, but this looks like the right fix.
> 
> Ewen Cheslack-Postava wrote:
> This happened consistently for me, which provided good motivation to 
> figure out how to fix it :) It looks like the tests use the default JVM 
> settings (build.gradle has overrides for compile, but not for test), so the 
> 500 MB allocation was always failing since it obviously exceeds the default 
> 128 MB heap. Now I'm wondering if other people are running with different 
> settings. Maybe if you have a gradle daemon running that was started with a 
> bigger max heap size?
> 
> Gwen Shapira wrote:
> My gradle.properties has:
> org.gradle.jvmargs=-XX:MaxPermSize=512m -Xmx1024m
> 
> I take it that yours doesn't? 
> 
> The builds in Apache's jenkins didn't seem to fail on that test either...

The one checked into the repo does, and I double checked that I'm not 
overriding that setting in my ~/.gradle/gradle.properties. But as far as I can 
tell that only applies to the gradle daemon process. It forks off another 
process to run the tests, which doesn't inherit those settings. If I add this 
to the build.gradle file next to the overrides for ScalaCompile tasks:

tasks.withType(JavaForkOptions) {
jvmArgs = ['-XX:MaxPermSize=512m', '-Xmx1024m']
}
  
then the forked process gets those settings and the test passes without this 
patch.


- Ewen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31447/#review74200
---


On Feb. 25, 2015, 11:27 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31447/
> ---
> 
> (Updated Feb. 25, 2015, 11:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1881
> https://issues.apache.org/jira/browse/KAFKA-1881
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1881: Fix out of memory error in testDeleteTopicWithCleaner by making 
> the dedupe buffer much smaller.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 0cbd72684a0cc6fe9f1fba2dc834de675b2eb994 
> 
> Diff: https://reviews.apache.org/r/31447/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

Re: Review Request 31447: Patch for KAFKA-1881

[Gwen Shapira]

> On Feb. 26, 2015, 3:07 a.m., Gwen Shapira wrote:
> > Ah, now I get what "dedupe buffer" refers to :)
> > 
> > I never managed to reproduce the oom, but this looks like the right fix.
> 
> Ewen Cheslack-Postava wrote:
> This happened consistently for me, which provided good motivation to 
> figure out how to fix it :) It looks like the tests use the default JVM 
> settings (build.gradle has overrides for compile, but not for test), so the 
> 500 MB allocation was always failing since it obviously exceeds the default 
> 128 MB heap. Now I'm wondering if other people are running with different 
> settings. Maybe if you have a gradle daemon running that was started with a 
> bigger max heap size?

My gradle.properties has:
org.gradle.jvmargs=-XX:MaxPermSize=512m -Xmx1024m

I take it that yours doesn't? 

The builds in Apache's jenkins didn't seem to fail on that test either...


- Gwen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31447/#review74200
---


On Feb. 25, 2015, 11:27 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31447/
> ---
> 
> (Updated Feb. 25, 2015, 11:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1881
> https://issues.apache.org/jira/browse/KAFKA-1881
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1881: Fix out of memory error in testDeleteTopicWithCleaner by making 
> the dedupe buffer much smaller.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 0cbd72684a0cc6fe9f1fba2dc834de675b2eb994 
> 
> Diff: https://reviews.apache.org/r/31447/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

Re: Review Request 31447: Patch for KAFKA-1881

[Ewen Cheslack-Postava]

> On Feb. 26, 2015, 3:07 a.m., Gwen Shapira wrote:
> > Ah, now I get what "dedupe buffer" refers to :)
> > 
> > I never managed to reproduce the oom, but this looks like the right fix.

This happened consistently for me, which provided good motivation to figure out 
how to fix it :) It looks like the tests use the default JVM settings 
(build.gradle has overrides for compile, but not for test), so the 500 MB 
allocation was always failing since it obviously exceeds the default 128 MB 
heap. Now I'm wondering if other people are running with different settings. 
Maybe if you have a gradle daemon running that was started with a bigger max 
heap size?


- Ewen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31447/#review74200
---


On Feb. 25, 2015, 11:27 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31447/
> ---
> 
> (Updated Feb. 25, 2015, 11:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1881
> https://issues.apache.org/jira/browse/KAFKA-1881
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1881: Fix out of memory error in testDeleteTopicWithCleaner by making 
> the dedupe buffer much smaller.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 0cbd72684a0cc6fe9f1fba2dc834de675b2eb994 
> 
> Diff: https://reviews.apache.org/r/31447/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

[jira] [Updated] (KAFKA-1929) Convert core kafka module to use the errors in org.apache.kafka.common.errors

[Jeff Holoman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Holoman updated KAFKA-1929:

Attachment: KAFKA-1929.patch

> Convert core kafka module to use the errors in org.apache.kafka.common.errors
> -
>
> Key: KAFKA-1929
> URL: https://issues.apache.org/jira/browse/KAFKA-1929
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>Assignee: Jeff Holoman
> Attachments: KAFKA-1929.patch
>
>
> With the introduction of the common package there are now a lot of errors 
> duplicated in both the common package and in the server. We should refactor 
> the server code (but not the scala clients) to switch over to the exceptions 
> in common.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1929) Convert core kafka module to use the errors in org.apache.kafka.common.errors

[Jeff Holoman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Holoman updated KAFKA-1929:

Status: Patch Available  (was: Open)

> Convert core kafka module to use the errors in org.apache.kafka.common.errors
> -
>
> Key: KAFKA-1929
> URL: https://issues.apache.org/jira/browse/KAFKA-1929
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>Assignee: Jeff Holoman
> Attachments: KAFKA-1929.patch
>
>
> With the introduction of the common package there are now a lot of errors 
> duplicated in both the common package and in the server. We should refactor 
> the server code (but not the scala clients) to switch over to the exceptions 
> in common.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1929) Convert core kafka module to use the errors in org.apache.kafka.common.errors

[Jeff Holoman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337828#comment-14337828
 ] 

Jeff Holoman commented on KAFKA-1929:
-

Created reviewboard https://reviews.apache.org/r/31460/diff/
 against branch origin/trunk

> Convert core kafka module to use the errors in org.apache.kafka.common.errors
> -
>
> Key: KAFKA-1929
> URL: https://issues.apache.org/jira/browse/KAFKA-1929
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>Assignee: Jeff Holoman
> Attachments: KAFKA-1929.patch
>
>
> With the introduction of the common package there are now a lot of errors 
> duplicated in both the common package and in the server. We should refactor 
> the server code (but not the scala clients) to switch over to the exceptions 
> in common.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31460: Patch for KAFKA-1929

[Jeff Holoman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31460/
---

Review request for kafka.


Bugs: KAFKA-1929
https://issues.apache.org/jira/browse/KAFKA-1929


Repository: kafka


Description
---

First Pass


Diffs
-

  
clients/src/main/java/org/apache/kafka/common/errors/LeaderNotAvailableException.java
 9d7ebd47a8439f38104ba62754227d4189418f62 
  
clients/src/main/java/org/apache/kafka/common/errors/OffsetMetadataTooLarge.java
 0be2f500685b09822aac3ccc9404bfacbbb34d17 
  clients/src/main/java/org/apache/kafka/common/errors/RetriableException.java 
6c639a972d7e4700fdbdcb32333832cbe8f991f3 
  clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
a8deac4ce5149129d0a6f44c0526af9d55649a36 
  core/src/main/scala/kafka/cluster/Partition.scala 
e6ad8be5e33b6fb31c078ad78f8de709869ddc04 
  core/src/main/scala/kafka/common/ErrorMapping.scala 
eedc2f5f21dd8755fba891998456351622e17047 
  core/src/main/scala/kafka/common/InvalidTopicException.scala 
59f887490d4172d7e89450a487dcbfabee73cb81 
  core/src/main/scala/kafka/common/NotEnoughReplicasAfterAppendException.scala 
c4f9def6162e9e25b273ca00b0974a4096cca041 
  core/src/main/scala/kafka/common/NotEnoughReplicasException.scala 
bfbe0ee4a5a15df69a94e7d7792bd11707787d92 
  core/src/main/scala/kafka/common/NotLeaderForPartitionException.scala 
b4558f89f0a23036c653aaba7f864c87fe952ae8 
  core/src/main/scala/kafka/common/OffsetMetadataTooLargeException.scala 
50edb273b3a799f8a8077a15329afc4ceda4abfb 
  core/src/main/scala/kafka/common/OffsetOutOfRangeException.scala 
0a2514cc0d99f05edf3b23dd79604708042e66ee 
  core/src/main/scala/kafka/common/Topic.scala 
ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
  core/src/main/scala/kafka/log/Log.scala 
846023bb98d0fa0603016466360c97071ac935ea 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
dd602ee2e65c2cd4ec363c75fa5d0b3c038b1ed2 
  core/src/main/scala/kafka/server/KafkaApis.scala 
6ee7d8819a9ef923f3a65c865a0a3d8ded8845f0 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
fb948b9ab28c516e81dab14dcbe211dcd99842b6 
  core/src/test/scala/other/kafka/StressTestLog.scala 
e19b8b28383554512a0c1f651d6764650d8db9c4 
  core/src/test/scala/unit/kafka/common/TopicTest.scala 
0fb25880c24adef906cd06359b624e7c8eb94ca6 
  core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala 
aeb7a19acaefabcc161c2ee6144a56d9a8999a81 
  core/src/test/scala/unit/kafka/log/LogManagerTest.scala 
90cd53033fafaf952ba0b3f1e28b0e1f1ad3ea76 
  core/src/test/scala/unit/kafka/log/LogTest.scala 
c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e 

Diff: https://reviews.apache.org/r/31460/diff/


Testing
---


Thanks,

Jeff Holoman

Re: Review Request 31306: Patch for KAFKA-1755

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31306/#review74201
---



core/src/main/scala/kafka/message/ByteBufferMessageSet.scala


Can we add the key-validation logic into analyzeAndValidateMessageSet()?



core/src/main/scala/kafka/message/ByteBufferMessageSet.scala


Why have bufferLimit if we already has sizeInBytes?


- Guozhang Wang


On Feb. 23, 2015, 10:29 p.m., Joel Koshy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31306/
> ---
> 
> (Updated Feb. 23, 2015, 10:29 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1755
> https://issues.apache.org/jira/browse/KAFKA-1755
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1755 v2
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/log/Log.scala 
> 846023bb98d0fa0603016466360c97071ac935ea 
>   core/src/main/scala/kafka/log/LogCleaner.scala 
> f8e7cd5fabce78c248a9027c4bb374a792508675 
>   core/src/main/scala/kafka/log/LogCleanerManager.scala 
> fd87d90597981c867a9b23731fca3b555bf85b7f 
>   core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 
> f46ad5cbbbad77d8d1f490d1f8aac97858da9b06 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 83d52643028c5628057dc0aa29819becfda61fdb 
>   core/src/test/scala/unit/kafka/log/CleanerTest.scala 
> d10e4f4ccbca5e50d81a243d3ab30cc7314b7fef 
>   core/src/test/scala/unit/kafka/log/LogTest.scala 
> c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e 
>   core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala 
> 73a26377eb63ab9989698e0491049434f032cba2 
> 
> Diff: https://reviews.apache.org/r/31306/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Joel Koshy
> 
>

Re: Review Request 31447: Patch for KAFKA-1881

[Gwen Shapira]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31447/#review74200
---

Ship it!


Ah, now I get what "dedupe buffer" refers to :)

I never managed to reproduce the oom, but this looks like the right fix.

- Gwen Shapira


On Feb. 25, 2015, 11:27 p.m., Ewen Cheslack-Postava wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31447/
> ---
> 
> (Updated Feb. 25, 2015, 11:27 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1881
> https://issues.apache.org/jira/browse/KAFKA-1881
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1881: Fix out of memory error in testDeleteTopicWithCleaner by making 
> the dedupe buffer much smaller.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
> 0cbd72684a0cc6fe9f1fba2dc834de675b2eb994 
> 
> Diff: https://reviews.apache.org/r/31447/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ewen Cheslack-Postava
> 
>

[jira] [Commented] (KAFKA-1400) transient unit test failure in SocketServerTest

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337652#comment-14337652
 ] 

Gwen Shapira commented on KAFKA-1400:
-

On second thought, why do we even try to detect from the client whether the 
server closed the connection? 

We have the server right there... we can just:
Assert(server.acceptor.socket.isClosed)

Any reason not to do this?

> transient unit test failure in SocketServerTest
> ---
>
> Key: KAFKA-1400
> URL: https://issues.apache.org/jira/browse/KAFKA-1400
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Jun Rao
>Assignee: Neha Narkhede
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-1400.patch
>
>
> Saw the following transient failure.
> kafka.network.SocketServerTest > testSocketsCloseOnShutdown FAILED 
> java.lang.AssertionError: Expected exception: java.net.SocketException 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 30763: KAFKA-1865: Producer Flush: Remove synchronization.

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30763/
---

(Updated Feb. 26, 2015, 1:16 a.m.)


Review request for kafka.


Bugs: KAFKA-1865
https://issues.apache.org/jira/browse/KAFKA-1865


Repository: kafka


Description
---

KAFKA-1865 Add a flush() method to the producer.


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/Metadata.java 
e8afecda956303a6ee116499fd443a54c018e17d 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1fd6917c8a5131254c740abad7f7228a47e3628c 
  clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
84530f2b948f9abd74203db48707e490dd9c81a5 
  clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
17fe541588d462c68c33f6209717cc4015e9b62f 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerRecord.java 
4990692efa6f01c62e1d7b05fbf31bec50e398c9 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/FutureRecordMetadata.java
 4a2da41f47994f778109e3c4107ffd90195f0bae 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 ecfe2144d778a5d9b614df5278b9f0a15637f10b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 dd0af8aee98abed5d4a0dc50989e37888bb353fe 
  clients/src/main/java/org/apache/kafka/common/errors/InterruptException.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/utils/SystemTime.java 
d682bd46ec3826f0a72388cc4ec30e1b1223d0f3 
  clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java 
4ae43ed47e31ad8052b4348a731da11120968508 
  clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
743aa7e523dd476949f484bfa4c7fb8a3afd7bf8 
  clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java 
75513b0bdd439329c5771d87436ef83fda853bfb 
  clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 
404bedb3dc4e44cc79251d71e1e3f8efdab60efa 
  
clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
 83338633717cfa4ef7cf2a590b5aa6b9c8cb1dd2 
  clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
558942aaecd1b9f7098435d39aa4b362cd16ff0a 
  core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
2802a399bf599e9530f53b7df72f12702a10d3c4 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
b15237b76def3b234924280fa3fdb25dbb0cc0dc 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
21d0ed2cb7c9459261d3cdc7c21dece5e2079698 

Diff: https://reviews.apache.org/r/30763/diff/


Testing (updated)
---

The latest patch uses Jiangjie's suggestion to remove the synchronization on 
flush.


Thanks,

Jay Kreps

[jira] [Commented] (KAFKA-1865) Add a flush() call to the new producer API

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337636#comment-14337636
 ] 

Jay Kreps commented on KAFKA-1865:
--

Updated reviewboard https://reviews.apache.org/r/30763/diff/
 against branch trunk

> Add a flush() call to the new producer API
> --
>
> Key: KAFKA-1865
> URL: https://issues.apache.org/jira/browse/KAFKA-1865
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1865.patch, KAFKA-1865_2015-02-21_15:36:54.patch, 
> KAFKA-1865_2015-02-22_16:26:46.patch, KAFKA-1865_2015-02-23_18:29:16.patch, 
> KAFKA-1865_2015-02-25_17:15:26.patch
>
>
> The postconditions of this would be that any record enqueued prior to flush() 
> would have completed being sent (either successfully or not).
> An open question is whether you can continue sending new records while this 
> call is executing (on other threads).
> We should only do this if it doesn't add inefficiencies for people who don't 
> use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1865) Add a flush() call to the new producer API

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1865:
-
Attachment: KAFKA-1865_2015-02-25_17:15:26.patch

> Add a flush() call to the new producer API
> --
>
> Key: KAFKA-1865
> URL: https://issues.apache.org/jira/browse/KAFKA-1865
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Attachments: KAFKA-1865.patch, KAFKA-1865_2015-02-21_15:36:54.patch, 
> KAFKA-1865_2015-02-22_16:26:46.patch, KAFKA-1865_2015-02-23_18:29:16.patch, 
> KAFKA-1865_2015-02-25_17:15:26.patch
>
>
> The postconditions of this would be that any record enqueued prior to flush() 
> would have completed being sent (either successfully or not).
> An open question is whether you can continue sending new records while this 
> call is executing (on other threads).
> We should only do this if it doesn't add inefficiencies for people who don't 
> use it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 30763: KAFKA-1865: Producer Flush: Remove synchronization.

[Jay Kreps]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30763/
---

(Updated Feb. 26, 2015, 1:15 a.m.)


Review request for kafka.


Summary (updated)
-

KAFKA-1865: Producer Flush: Remove synchronization.


Bugs: KAFKA-1865
https://issues.apache.org/jira/browse/KAFKA-1865


Repository: kafka


Description
---

KAFKA-1865 Add a flush() method to the producer.


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/Metadata.java 
e8afecda956303a6ee116499fd443a54c018e17d 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
1fd6917c8a5131254c740abad7f7228a47e3628c 
  clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
84530f2b948f9abd74203db48707e490dd9c81a5 
  clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
17fe541588d462c68c33f6209717cc4015e9b62f 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerRecord.java 
4990692efa6f01c62e1d7b05fbf31bec50e398c9 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/FutureRecordMetadata.java
 4a2da41f47994f778109e3c4107ffd90195f0bae 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 ecfe2144d778a5d9b614df5278b9f0a15637f10b 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 dd0af8aee98abed5d4a0dc50989e37888bb353fe 
  clients/src/main/java/org/apache/kafka/common/errors/InterruptException.java 
PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/utils/SystemTime.java 
d682bd46ec3826f0a72388cc4ec30e1b1223d0f3 
  clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java 
4ae43ed47e31ad8052b4348a731da11120968508 
  clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
743aa7e523dd476949f484bfa4c7fb8a3afd7bf8 
  clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java 
75513b0bdd439329c5771d87436ef83fda853bfb 
  clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 
404bedb3dc4e44cc79251d71e1e3f8efdab60efa 
  
clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
 83338633717cfa4ef7cf2a590b5aa6b9c8cb1dd2 
  clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
558942aaecd1b9f7098435d39aa4b362cd16ff0a 
  core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
2802a399bf599e9530f53b7df72f12702a10d3c4 
  core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
b15237b76def3b234924280fa3fdb25dbb0cc0dc 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
21d0ed2cb7c9459261d3cdc7c21dece5e2079698 

Diff: https://reviews.apache.org/r/30763/diff/


Testing
---

New patch addresses feedback. Also (1) comments out the consumer tests so I 
could verify everything else passes and (2) moves some unit tests I found that 
were in the wrong packages.


Thanks,

Jay Kreps

Re: Review Request 30763: KAFKA-1865: Producer Flush: if at first your patch doesn't succeed, try, try, try again.

[Jay Kreps]

> On Feb. 24, 2015, 5:20 p.m., Guozhang Wang wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java, 
> > line 379
> > 
> >
> > Can also throw SchemaException.

Yeah I intentionally left this and IllegalArgumentException our of the docs 
since they basically indicate "programmer error" so you aren't expected to 
handle them, just crash and fix your code.


> On Feb. 24, 2015, 5:20 p.m., Guozhang Wang wrote:
> > clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java,
> >  line 91
> > 
> >
> > I still think it is kind of waste to create duplicate RecordMetadata 
> > here and in done()..

I take your point, but I tried the refactor to remove it and it wasn't very 
simple plus I think it would be quite rare that you would ever actually invoke 
both paths. Plus optimizing adds another synchronous memory reference while 
these super quick object allocations may get optimized out anyway...


- Jay


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30763/#review73829
---


On Feb. 24, 2015, 2:31 a.m., Jay Kreps wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30763/
> ---
> 
> (Updated Feb. 24, 2015, 2:31 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1865
> https://issues.apache.org/jira/browse/KAFKA-1865
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1865 Add a flush() method to the producer.
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/Metadata.java 
> e8afecda956303a6ee116499fd443a54c018e17d 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 1fd6917c8a5131254c740abad7f7228a47e3628c 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 84530f2b948f9abd74203db48707e490dd9c81a5 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 17fe541588d462c68c33f6209717cc4015e9b62f 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerRecord.java 
> 4990692efa6f01c62e1d7b05fbf31bec50e398c9 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/FutureRecordMetadata.java
>  4a2da41f47994f778109e3c4107ffd90195f0bae 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  ecfe2144d778a5d9b614df5278b9f0a15637f10b 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  dd0af8aee98abed5d4a0dc50989e37888bb353fe 
>   
> clients/src/main/java/org/apache/kafka/common/errors/InterruptException.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/utils/SystemTime.java 
> d682bd46ec3826f0a72388cc4ec30e1b1223d0f3 
>   clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java 
> 4ae43ed47e31ad8052b4348a731da11120968508 
>   clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
> 743aa7e523dd476949f484bfa4c7fb8a3afd7bf8 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java 
> 75513b0bdd439329c5771d87436ef83fda853bfb 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 
> 29c8417422c0cf0d29bf2405c77fd05e35350259 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
>  83338633717cfa4ef7cf2a590b5aa6b9c8cb1dd2 
>   clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
> 558942aaecd1b9f7098435d39aa4b362cd16ff0a 
>   core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
> 2802a399bf599e9530f53b7df72f12702a10d3c4 
>   core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 
> b15237b76def3b234924280fa3fdb25dbb0cc0dc 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
> 21d0ed2cb7c9459261d3cdc7c21dece5e2079698 
> 
> Diff: https://reviews.apache.org/r/30763/diff/
> 
> 
> Testing
> ---
> 
> New patch addresses feedback. Also (1) comments out the consumer tests so I 
> could verify everything else passes and (2) moves some unit tests I found 
> that were in the wrong packages.
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>

Re: Unit tests in java7 vs java8

[Guozhang Wang]

I am working on the test handing / NPE / failure issues of ConsumerTest
only as part of the consumer rewriting (KAFKA-1910).

I currently run Java 8 and the unit test takes about 10 minutes, I can do
"time ./gradlew test" with both versions and see if there is a clear
difference.

Guozhang

On Wed, Feb 25, 2015 at 4:53 PM, Jay Kreps  wrote:

> Yeah, hey Guozhang, is that fix part of the larger consumer patch you just
> posted or is that a separate issue?
>
> -Jay
>
> On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira 
> wrote:
>
> > The Consumer tests are currently hanging :(
> >
> > I think Guozhang is working on a solution. I'm commenting them out until
> > the problem is resolved...
> >
> >
> >
> > On Wed, Feb 25, 2015 at 4:00 PM, Tong Li  wrote:
> >
> > > Gwen,
> > > I have not tried Java 8. Still on Java 7, but I always run into the
> > > test hung problems (no errors on the screen and the system is
> completely
> > > idle), it may be a different problem. I can recreate that problem every
> > > time when I run "gradle --daemon testAll", I recall that couple of
> weeks
> > > ago there was one patch saying fixed the problem, but I am still seeing
> > the
> > > problem with latest code. What I noticed is that seems tests always
> stop
> > at
> > > one of the ConsumerTest test cases. What puzzled me the most is that it
> > was
> > > not always a particular test case. Being very new in this community, I
> > > think that error must be something related to my env. Here is my
> > > environment:
> > >
> > >  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
> > > big enough max lock memory,
> > >
> > > not complaining, just some observations in case you wonder what other
> > > developers may face.
> > >
> > > Thanks.
> > >
> > > Tong Li
> > > OpenStack & Kafka Community Development
> > > Building 501/B205
> > > liton...@us.ibm.com
> > >
> > > [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
> > > PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
> > > Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java
> 8? I
> > > just noticed that they take almost twice
> > >
> > > From: Gwen Shapira 
> > > To: "dev@kafka.apache.org" 
> > > Date: 02/25/2015 03:47 PM
> > > Subject: Unit tests in java7 vs java8
> > > --
> > >
> > >
> > >
> > > Hi,
> > >
> > > Anyone running tests on Java 8? I just noticed that they take almost
> > twice
> > > as long to run compared to Java 7 (at least on my box, and with Scala
> > > 2.10.4).
> > >
> > > Anyone else noticed this? Maybe even did some digging on the causes?
> > >
> > > Gwen
> > >
> > >
> >
>



-- 
-- Guozhang

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337621#comment-14337621
 ] 

Jay Kreps commented on KAFKA-1910:
--

Where is the RB?

> Refactor KafkaConsumer
> --
>
> Key: KAFKA-1910
> URL: https://issues.apache.org/jira/browse/KAFKA-1910
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> KafkaConsumer now contains all the logic on the consumer side, making it a 
> very huge class file, better re-factoring it to have multiple layers on top 
> of KafkaClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Unit tests in java7 vs java8

[Guozhang Wang]

I am working on the test handing / NPE / failure issues of ConsumerTest only.

I currently run Java 8 and the unit test takes about 10 minutes, I can do "time 
./gradlew test" with both versions and see if there is a clear difference.

Guozhang

From: Jay Kreps [jay.kr...@gmail.com]
Sent: Wednesday, February 25, 2015 4:53 PM
To: dev@kafka.apache.org; Guozhang Wang
Subject: Re: Unit tests in java7 vs java8

Yeah, hey Guozhang, is that fix part of the larger consumer patch you just 
posted or is that a separate issue?

-Jay

On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira 
mailto:gshap...@cloudera.com>> wrote:
The Consumer tests are currently hanging :(

I think Guozhang is working on a solution. I'm commenting them out until
the problem is resolved...



On Wed, Feb 25, 2015 at 4:00 PM, Tong Li 
mailto:liton...@us.ibm.com>> wrote:

> Gwen,
> I have not tried Java 8. Still on Java 7, but I always run into the
> test hung problems (no errors on the screen and the system is completely
> idle), it may be a different problem. I can recreate that problem every
> time when I run "gradle --daemon testAll", I recall that couple of weeks
> ago there was one patch saying fixed the problem, but I am still seeing the
> problem with latest code. What I noticed is that seems tests always stop at
> one of the ConsumerTest test cases. What puzzled me the most is that it was
> not always a particular test case. Being very new in this community, I
> think that error must be something related to my env. Here is my
> environment:
>
>  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
> big enough max lock memory,
>
> not complaining, just some observations in case you wonder what other
> developers may face.
>
> Thanks.
>
> Tong Li
> OpenStack & Kafka Community Development
> Building 501/B205
> liton...@us.ibm.com
>
> [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
> PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
> Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I
> just noticed that they take almost twice
>
> From: Gwen Shapira mailto:gshap...@cloudera.com>>
> To: "dev@kafka.apache.org" 
> mailto:dev@kafka.apache.org>>
> Date: 02/25/2015 03:47 PM
> Subject: Unit tests in java7 vs java8
> --
>
>
>
> Hi,
>
> Anyone running tests on Java 8? I just noticed that they take almost twice
> as long to run compared to Java 7 (at least on my box, and with Scala
> 2.10.4).
>
> Anyone else noticed this? Maybe even did some digging on the causes?
>
> Gwen
>
>

[jira] [Reopened] (KAFKA-1400) transient unit test failure in SocketServerTest

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira reopened KAFKA-1400:
-

> transient unit test failure in SocketServerTest
> ---
>
> Key: KAFKA-1400
> URL: https://issues.apache.org/jira/browse/KAFKA-1400
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Jun Rao
>Assignee: Neha Narkhede
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-1400.patch
>
>
> Saw the following transient failure.
> kafka.network.SocketServerTest > testSocketsCloseOnShutdown FAILED 
> java.lang.AssertionError: Expected exception: java.net.SocketException 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1400) transient unit test failure in SocketServerTest

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337608#comment-14337608
 ] 

Gwen Shapira commented on KAFKA-1400:
-

Kinda strange, the TCPDUMP looks the same whether I get an exception or not:

{code}
16:52:17.249207 IP localhost.55538 > localhost.9095: Flags [S], seq 3746193634, 
win 65535, options [mss 16344,nop,wscale 5,nop,nop,TS val 1314712608 ecr 
0,sackOK,eol], length 0
16:52:17.249257 IP localhost.9095 > localhost.55538: Flags [S.], seq 999508519, 
ack 3746193635, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 
1314712608 ecr 1314712608,sackOK,eol], length 0
16:52:17.249264 IP localhost.55538 > localhost.9095: Flags [.], ack 1, win 
12759, options [nop,nop,TS val 1314712608 ecr 1314712608], length 0
16:52:17.249270 IP localhost.9095 > localhost.55538: Flags [.], ack 1, win 
38788, options [nop,nop,TS val 1314712608 ecr 1314712608], length 0
16:52:22.436127 IP localhost.55538 > localhost.9095: Flags [P.], seq 1:2, ack 
1, win 12759, options [nop,nop,TS val 1314717768 ecr 1314712608], length 1
16:52:22.436162 IP localhost.9095 > localhost.55538: Flags [.], ack 2, win 
38788, options [nop,nop,TS val 1314717768 ecr 1314717768], length 0
16:52:22.436169 IP localhost.55538 > localhost.9095: Flags [P.], seq 2:7, ack 
1, win 12759, options [nop,nop,TS val 1314717768 ecr 1314717768], length 5
16:52:22.436173 IP localhost.9095 > localhost.55538: Flags [.], ack 7, win 
38787, options [nop,nop,TS val 1314717768 ecr 1314717768], length 0
16:52:22.436177 IP localhost.55538 > localhost.9095: Flags [P.], seq 7:47, ack 
1, win 12759, options [nop,nop,TS val 1314717768 ecr 1314717768], length 40
16:52:22.436180 IP localhost.9095 > localhost.55538: Flags [.], ack 47, win 
38782, options [nop,nop,TS val 1314717768 ecr 1314717768], length 0
16:52:22.484320 IP localhost.9095 > localhost.55538: Flags [P.], seq 1:23, ack 
47, win 38782, options [nop,nop,TS val 1314717808 ecr 1314717768], length 22
16:52:22.484353 IP localhost.55538 > localhost.9095: Flags [.], ack 23, win 
12758, options [nop,nop,TS val 1314717808 ecr 1314717808], length 0
16:52:22.552365 IP localhost.9095 > localhost.55538: Flags [F.], seq 23, ack 
47, win 38782, options [nop,nop,TS val 1314717872 ecr 1314717808], length 0
16:52:22.552372 IP localhost.55538 > localhost.9095: Flags [.], ack 24, win 
12758, options [nop,nop,TS val 1314717872 ecr 1314717872], length 0
16:52:22.552376 IP localhost.9095 > localhost.55538: Flags [.], ack 47, win 
38782, options [nop,nop,TS val 1314717872 ecr 1314717872], length 0
16:52:28.632196 IP localhost.55538 > localhost.9095: Flags [P.], seq 47:48, ack 
24, win 12758, options [nop,nop,TS val 1314723916 ecr 1314717872], length 1
16:52:28.632235 IP localhost.9095 > localhost.55538: Flags [R], seq 999508543, 
win 0, length 0
{code}

I get a FIN, I ack it. I send one extra packet and I get a RST. Pretty much as 
expected.

Whether I get an exception or not, OTOH is completely random. I'm guessing this 
is because the .write() and .flush() methods are async.

As documented for output stream: The flush method of 
OutputStream does nothing.

I *think* that if I force a sleep after some bytes were written, I'll always 
get an exception. I'm not too happy with this approach.

Any thoughts?



> transient unit test failure in SocketServerTest
> ---
>
> Key: KAFKA-1400
> URL: https://issues.apache.org/jira/browse/KAFKA-1400
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Jun Rao
>Assignee: Neha Narkhede
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-1400.patch
>
>
> Saw the following transient failure.
> kafka.network.SocketServerTest > testSocketsCloseOnShutdown FAILED 
> java.lang.AssertionError: Expected exception: java.net.SocketException 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Unit tests in java7 vs java8

[Jay Kreps]

Yeah, hey Guozhang, is that fix part of the larger consumer patch you just
posted or is that a separate issue?

-Jay

On Wed, Feb 25, 2015 at 4:25 PM, Gwen Shapira  wrote:

> The Consumer tests are currently hanging :(
>
> I think Guozhang is working on a solution. I'm commenting them out until
> the problem is resolved...
>
>
>
> On Wed, Feb 25, 2015 at 4:00 PM, Tong Li  wrote:
>
> > Gwen,
> > I have not tried Java 8. Still on Java 7, but I always run into the
> > test hung problems (no errors on the screen and the system is completely
> > idle), it may be a different problem. I can recreate that problem every
> > time when I run "gradle --daemon testAll", I recall that couple of weeks
> > ago there was one patch saying fixed the problem, but I am still seeing
> the
> > problem with latest code. What I noticed is that seems tests always stop
> at
> > one of the ConsumerTest test cases. What puzzled me the most is that it
> was
> > not always a particular test case. Being very new in this community, I
> > think that error must be something related to my env. Here is my
> > environment:
> >
> >  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
> > big enough max lock memory,
> >
> > not complaining, just some observations in case you wonder what other
> > developers may face.
> >
> > Thanks.
> >
> > Tong Li
> > OpenStack & Kafka Community Development
> > Building 501/B205
> > liton...@us.ibm.com
> >
> > [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
> > PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
> > Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I
> > just noticed that they take almost twice
> >
> > From: Gwen Shapira 
> > To: "dev@kafka.apache.org" 
> > Date: 02/25/2015 03:47 PM
> > Subject: Unit tests in java7 vs java8
> > --
> >
> >
> >
> > Hi,
> >
> > Anyone running tests on Java 8? I just noticed that they take almost
> twice
> > as long to run compared to Java 7 (at least on my box, and with Scala
> > 2.10.4).
> >
> > Anyone else noticed this? Maybe even did some digging on the causes?
> >
> > Gwen
> >
> >
>

[jira] [Commented] (KAFKA-1460) NoReplicaOnlineException: No replica for partition

[Chad Retz (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337575#comment-14337575
 ] 

Chad Retz commented on KAFKA-1460:
--

Likewise, embedded Kafka shutdowns (literally KafkaServer.shutdown) is causing 
this for me as well.

> NoReplicaOnlineException: No replica for partition
> --
>
> Key: KAFKA-1460
> URL: https://issues.apache.org/jira/browse/KAFKA-1460
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Artur Denysenko
>Priority: Critical
> Attachments: state-change.log
>
>
> We have a standalone kafka server.
> After several days of running we get:
> {noformat}
> kafka.common.NoReplicaOnlineException: No replica for partition 
> [gk.q.module,1] is alive. Live brokers are: [Set()], Assigned replicas are: 
> [List(0)]
>   at 
> kafka.controller.OfflinePartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:61)
>   at 
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:336)
>   at 
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:185)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:99)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:96)
>   at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>   at 
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
>   at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
>   at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742)
>   at 
> kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:96)
>   at 
> kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:68)
>   at 
> kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:312)
>   at 
> kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:162)
>   at 
> kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:63)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply$mcZ$sp(KafkaController.scala:1068)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1066)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1066)
>   at kafka.utils.Utils$.inLock(Utils.scala:538)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener.handleNewSession(KafkaController.scala:1066)
>   at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
>   at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}
> Please see attached [state-change.log]
> You can find all server logs (450mb) here: 
> http://46.4.114.35:/deploy/kafka-logs.2014-05-14-16.tgz
> On client we get:
> {noformat}
> 16:28:36,843 [ool-12-thread-2] WARN  ZookeeperConsumerConnector - 
> [dev_dev-1400257716132-e7b8240c], no brokers found when trying to rebalance.
> {noformat}
> If we try to send message using 'kafka-console-producer.sh':
> {noformat}
> [root@dev kafka]# /srv/kafka/bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic test
> message
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
> details.
> [2014-05-16 19:45:30,950] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(test)] from broker [id:0,host:localhost,port:9092] failed 
> (kafka.client.ClientUtils$)
> java.net.SocketTimeoutException
> at 
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at 
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> at kafka.utils.Utils$.read(Utils.scala:375)
> at 
> kafka.network.BoundedByteBufferReceive.readFrom(B

[jira] [Commented] (KAFKA-1978) Replication test_0131 system test has been failing.

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337564#comment-14337564
 ] 

Ewen Cheslack-Postava commented on KAFKA-1978:
--

[~anigam] Can you explain the reasoning here? I'm not clear on why the current 
configuration should be considered incorrect. The last time I took a pass at 
cleaning up the system tests these issues cropped up. I filed KAFKA-1771 and 
that bug discusses some of the issues I found in the code, including the 
condition that triggers the bounds exception. Your patch fixes the issue by 
making sure replica_factor >= num_partitions, but I'm not sure why that 
condition should be required.

If it is just a config issue, you might want to check testcase_0132 and 
testcase_0133 since KAFKA-1771 says they have the same issue.

> Replication test_0131 system test has been failing.
> ---
>
> Key: KAFKA-1978
> URL: https://issues.apache.org/jira/browse/KAFKA-1978
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Abhishek Nigam
>Assignee: Abhishek Nigam
> Attachments: KAFKA-1978.patch
>
>
> Issue is an out of bounds exception due to mis-configuration of the test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Unit tests in java7 vs java8

[Gwen Shapira]

The Consumer tests are currently hanging :(

I think Guozhang is working on a solution. I'm commenting them out until
the problem is resolved...



On Wed, Feb 25, 2015 at 4:00 PM, Tong Li  wrote:

> Gwen,
> I have not tried Java 8. Still on Java 7, but I always run into the
> test hung problems (no errors on the screen and the system is completely
> idle), it may be a different problem. I can recreate that problem every
> time when I run "gradle --daemon testAll", I recall that couple of weeks
> ago there was one patch saying fixed the problem, but I am still seeing the
> problem with latest code. What I noticed is that seems tests always stop at
> one of the ConsumerTest test cases. What puzzled me the most is that it was
> not always a particular test case. Being very new in this community, I
> think that error must be something related to my env. Here is my
> environment:
>
>  Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles and
> big enough max lock memory,
>
> not complaining, just some observations in case you wonder what other
> developers may face.
>
> Thanks.
>
> Tong Li
> OpenStack & Kafka Community Development
> Building 501/B205
> liton...@us.ibm.com
>
> [image: Inactive hide details for Gwen Shapira ---02/25/2015 03:47:58
> PM---Hi, Anyone running tests on Java 8? I just noticed that they]Gwen
> Shapira ---02/25/2015 03:47:58 PM---Hi, Anyone running tests on Java 8? I
> just noticed that they take almost twice
>
> From: Gwen Shapira 
> To: "dev@kafka.apache.org" 
> Date: 02/25/2015 03:47 PM
> Subject: Unit tests in java7 vs java8
> --
>
>
>
> Hi,
>
> Anyone running tests on Java 8? I just noticed that they take almost twice
> as long to run compared to Java 7 (at least on my box, and with Scala
> 2.10.4).
>
> Anyone else noticed this? Maybe even did some digging on the causes?
>
> Gwen
>
>

Re: Tips for working with Kafka and data streams

[Tong Li]

+2, these kind of articles coming from the ones who created Kafka always
provide great value to Kafka users and developers. For my 2 cents, I would
love to see one or two articles for developers who involved in Kafka
development on the topics of how to develop test cases and how to run them,
what to expect when error occurs, typical system settings, I suspect that
most of us do run it on linux based systems, little pointer probably can
help a lot. and most importantly how to set up your dev environment so that
you are not struggling with the things the pioneers have already figured
out. For example, recommended dev. ide, debug methods, of course, these
will be the preference of the writer, no one is obligated to use but can
certainly get people started quicker. As Kafka draw more interest, I
suspect more developers will join, having something like that can be
extremely helpful.

Jay, articles similar to the one linked in your original email can actually
be submitted to developerworks, and you can get some money out of it if you
like. If you do not know how to do that, I can certainly provide some
pointers if you are interested.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jay Kreps 
To: "dev@kafka.apache.org" ,
"us...@kafka.apache.org" 
Date:   02/25/2015 02:52 PM
Subject:Tips for working with Kafka and data streams



Hey guys,

One thing we tried to do along with the product release was start to put
together a practical guide for using Kafka. I wrote this up here:
http://blog.confluent.io/2015/02/25/stream-data-platform-1/

I'd like to keep expanding on this as good practices emerge and we learn
more stuff. So two questions:
1. Anything you think other people should know about working with data
streams? What did you wish you knew when you got started?
2. Anything you don't know about but would like to hear more about?

-Jay

Re: Review Request 31366: Patch for KAFKA-1461

[Eric Olander]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31366/#review74160
---



core/src/main/scala/kafka/server/AbstractFetcherThread.scala


Using the foreach method on partitionMap.get(topicAndPartition) would 
accomplish the same thing (lines 114-116, and 164) without the need for pattern 
matching.



core/src/main/scala/kafka/server/AbstractFetcherThread.scala


Again, foreach would be more idomatic, or take advantage of already being 
in a for loop.


- Eric Olander


On Feb. 24, 2015, 6:02 p.m., Sriharsha Chintalapani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31366/
> ---
> 
> (Updated Feb. 24, 2015, 6:02 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1461
> https://issues.apache.org/jira/browse/KAFKA-1461
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1461. Replica fetcher thread does not implement any back-off behavior.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/AbstractFetcherManager.scala 
> 20c00cb8cc2351950edbc8cb1752905a0c26e79f 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> 8c281d4668f92eff95a4a5df3c03c4b5b20e7095 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 14bf3216bae030331bdf76b3266ed0e73526c3de 
>   core/src/main/scala/kafka/server/OffsetAndDelay.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
> 6879e730282185bda3d6bc3659cb15af0672cecf 
>   core/src/test/scala/unit/kafka/server/ReplicaFetchTest.scala 
> da4bafc1e2a94a436efe395aab1888fc21e55748 
> 
> Diff: https://reviews.apache.org/r/31366/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sriharsha Chintalapani
> 
>

Re: Unit tests in java7 vs java8

[Tong Li]

Gwen,
I have not tried Java 8. Still on Java 7, but I always run into the
test hung problems (no errors on the screen and the system is completely
idle), it may be a different problem. I can recreate that problem every
time when I run "gradle --daemon testAll", I recall that couple of weeks
ago there was one patch saying fixed the problem, but I am still seeing the
problem with latest code. What I noticed is that seems tests always stop at
one of the ConsumerTest test cases. What puzzled me the most is that it was
not always a particular test case. Being very new in this community, I
think that error must be something related to my env. Here is my
environment:

Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles
and big enough max lock memory,

not complaining, just some observations in case you wonder what other
developers may face.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Gwen Shapira 
To: "dev@kafka.apache.org" 
Date:   02/25/2015 03:47 PM
Subject:Unit tests in java7 vs java8



Hi,

Anyone running tests on Java 8? I just noticed that they take almost twice
as long to run compared to Java 7 (at least on my box, and with Scala
2.10.4).

Anyone else noticed this? Maybe even did some digging on the causes?

Gwen

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

[Jiangjie Qin]

For 1), the current design allow you to do it. The customizable message
handler takes in a ConsumerRecord and spit a List, you can
just put a topic for the ProducerRecord different from ConsumerRecord.

WRT performance, we did some test in LinkedIn, the performance looks good
to us.

Jiangjie (Becket) Qin

On 2/25/15, 3:41 PM, "Bhavesh Mistry"  wrote:

>Hi Jiangjie,
>
>It might be too late.  But, I wanted to bring-up following use case for
>adopting new MM:
>
>1) Ability to publish message from src topic to different destination
>topic
>via --overidenTopics=srcTopic:newDestinationTopic
>
>In order to adopt, new MM enhancement customer will compare performance of
>new MM and data quality while running  old MM against same destination
>cluster in Prod.
>
>Let me know if you agree to that or not.  Also, If yes, will be able to
>able to provide this feature in release version.
>
>Thanks,
>
>Bhavesh
>
>
>On Tue, Feb 24, 2015 at 5:31 PM, Jiangjie Qin 
>wrote:
>
>> Sure! Just created the voting thread :)
>>
>> On 2/24/5, 4:44 PM, "Jay Kreps"  wrote:
>>
>> >Hey Jiangjie,
>> >
>> >Let's do an official vote so that we know what we are voting on and we
>>are
>> >crisp on what the outcome was. This thread is very long :-
>> >
>> >-Jay
>> >
>> >On Tue, Feb 24, 2015 at 2:53 PM, Jiangjie Qin
>>
>> >wrote:
>> >
>> >> I updated the KIP page based on the discussion we had.
>> >>
>> >> Should I launch another vote or we can think of this mail thread has
>> >> already included a vote?
>> >>
>> >> Jiangjie (Becket) Qin
>> >>
>> >> On 2/11/15, 5:15 PM, "Neha Nakhede"  wrote:
>> >>
>> >> >Thanks for the explanation, Joel! Would love to see the results of
>>the
>> >> >throughput experiment and I'm a +1 on everything els, ncluding the
>> >> >rebalance callback and record handler.
>> >> >
>> >> >-Neha
>> >> >
>> >> >On Wed, Feb 11, 2015 at 1:13 PM Jay Kreps 
>>wrote:
>> >> >
>> >> >> Cool, I agree with all that.
>> >> >>
>> >> >> I agree about the need for a rebalancing callback.
>> >> >>
>> >> >> Totally agree about record handler.
> >> >>
>> >> >> It would be great to see if a prototype of this is workable.
>> >> >>
>> >> >> Thanks guys!
>> >> >>
>> >> >> -Jay
>> >> >>
>> >> >> On Wed, Feb 11, 2015 at 12:36 PM, Joel Koshy 
>> >> >>wrote:
>> >> >>
>> >> >> > Hey Jay,
>> >> >> >
>> >> >> > Guozhang, Becket and I got together to discus this and we
>>think:
>> >> >> >
>> >> >> > - It seems that your proposal based on the new consumr and flush
>> >>call
>> >> >> >   should work.
>> >> >> > - We would likely need to call the poll with a timeout that
>>matches
>> >> >> >   the offset commit interval in ordr to deal with low volume
>> >> >> >   mirroring pipelines.
>> >> >> > - We will still need a rebalnce callback to reduce duplicates -
>> >>the
>> >> >> >   rebalance callback would need to flush and commit offsets.
>> >> >> > - The only remaining question is if the overall throughput is
>> >> >> >   sufficient. I think someone at LinkedIn (I don't remember who)
>> >>did
>> >> >> >   some experiments with data channel size == 1 and ran into
>>issues.
>> >> >> >   That was not thoroughly investigated though.
>> >> >> > - The addition of flush may actually make this solution viable
>>for
>> >>the
>> >> >> >   current mirror-maker (wih the old consumer). We can prototype
>> >>that
>> >> >> >   offline and if it works out well we can redo KAFKA-1650 (i.e.,
>> >> >> >   refactor the current mirror maker). The flush call and the new
>> >> >> >   consumer didn't exist at the time we did KAFKA-1650 so this
>>did
>> >>not
>> >> >> >   occur to us.
>> >> >> > - We think the RecordHandler is still a useful small addition
>>for
>> >>the
>> >> >> >   use-cases mentioned earlier in this thread.
>> >> >> >
>> >> >> > Thanks,
>> >> >> >
>> >> >> > Joel
>> >> >> >
>> >> >> > On Wed, Feb 11, 2015 at 09:05:39AM -0800, Jay Kreps wrote:
>> >> >> > > Guozhang, I agree with 1-3, I do think what I was proposing
>>was
>> >> >>simpler
>> >> >> > but
>> >> >> > > perhaps there re gaps in that?
>> >> >> > >
>> >> >> > > Hey Joel--Here was a sketch of what I was proposing. I do
>>think
>> >>this
>> >> >> > get's
>> >> >> > > rid of manual offset tracking, espcially doing so across
>>threads
>> >> >>with
>> >> >> > > dedicated commit threads, which I think is prety complex.
>> >> >> > >
>> >> >> > > while(true) {
>> >> >> > > val recs = consumer.poll(Long.MaxValue);
>> >> >> > > for (rec <- recs)
>> >> >> > > producer.snd(rec, logErrorCallback)
>> >> >> > > if(System.currentTimeMillis - lastCommit >
>>commitInterval) {
>> >> >> > > producer.flush()
>> >> >> > > consumer.commit()
>> >> >> > > lastCommit = System.currentTimeMillis
>> >> >> > > }
>> >> >> > > }
>> >> >> > >
>> >> >> > > (See the previous email for details). I think the question
>>is: is
>> >> >>there
>> >> >> > any
>> >> >> > > reason--performance, correctness, etc--that this won't work?
>> >> >>Basically
>> >> >> I
>> >> >> > > think you guys have thoug

Re: Review Request 29467: Patch for KAFKA-1660

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29467/#review74149
---


Could we add some unit tests for this new API, both called by another caller 
thread as well as in the callback. Things to check:

1. Concurrent calls to close().
2. Timeout value is honered in close().
3. If producer is closed immediately no buffered messages will get sent.


clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


I think we will use debug level logging here indicating function returning:


https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Error+Handling+and+Logging



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


Include the timeout value in the log entry?



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java


Ditto.


- Guozhang Wang


On Feb. 18, 2015, 12:41 a.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29467/
> ---
> 
> (Updated Feb. 18, 2015, 12:41 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1660
> https://issues.apache.org/jira/browse/KAFKA-1660
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Merge remote-tracking branch 'origin/trunk' into KAFKA-1660
> 
> Conflicts:
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 1fd6917c8a5131254c740abad7f7228a47e3628c 
>   clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
> 84530f2b948f9abd74203db48707e490dd9c81a5 
>   clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
> 17fe541588d462c68c33f6209717cc4015e9b62f 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> ed9c63a6679e3aaf83d19fde19268553a4c107c2 
> 
> Diff: https://reviews.apache.org/r/29467/diff/
> 
> 
> Testing
> ---
> 
> existing unit tests passed.
> 
> 
> Thanks,
> 
> Parth Brahmbhatt
> 
>

[jira] [Updated] (KAFKA-1986) Producer request failure rate should not include InvalidMessageSizeException and OffsetOutOfRangeException

[Aditya Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Auradkar updated KAFKA-1986:
---
Status: Patch Available  (was: Open)

> Producer request failure rate should not include InvalidMessageSizeException 
> and OffsetOutOfRangeException
> --
>
> Key: KAFKA-1986
> URL: https://issues.apache.org/jira/browse/KAFKA-1986
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
> Attachments: KAFKA-1986.patch
>
>
> InvalidMessageSizeException and OffsetOutOfRangeException should not be 
> counted a failedProduce and failedFetch requests since they are client side 
> errors. They should be treated similarly to UnknownTopicOrPartitionException 
> or NotLeaderForPartitionException



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1986) Producer request failure rate should not include InvalidMessageSizeException and OffsetOutOfRangeException

[Aditya A Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337506#comment-14337506
 ] 

Aditya A Auradkar commented on KAFKA-1986:
--

Created reviewboard https://reviews.apache.org/r/31449/diff/
 against branch origin/trunk

> Producer request failure rate should not include InvalidMessageSizeException 
> and OffsetOutOfRangeException
> --
>
> Key: KAFKA-1986
> URL: https://issues.apache.org/jira/browse/KAFKA-1986
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
> Attachments: KAFKA-1986.patch
>
>
> InvalidMessageSizeException and OffsetOutOfRangeException should not be 
> counted a failedProduce and failedFetch requests since they are client side 
> errors. They should be treated similarly to UnknownTopicOrPartitionException 
> or NotLeaderForPartitionException



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1986) Producer request failure rate should not include InvalidMessageSizeException and OffsetOutOfRangeException

[Aditya A Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya A Auradkar updated KAFKA-1986:
-
Attachment: KAFKA-1986.patch

> Producer request failure rate should not include InvalidMessageSizeException 
> and OffsetOutOfRangeException
> --
>
> Key: KAFKA-1986
> URL: https://issues.apache.org/jira/browse/KAFKA-1986
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
> Attachments: KAFKA-1986.patch
>
>
> InvalidMessageSizeException and OffsetOutOfRangeException should not be 
> counted a failedProduce and failedFetch requests since they are client side 
> errors. They should be treated similarly to UnknownTopicOrPartitionException 
> or NotLeaderForPartitionException



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31449: Patch for KAFKA-1986

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31449/
---

Review request for kafka.


Bugs: KAFKA-1986
https://issues.apache.org/jira/browse/KAFKA-1986


Repository: kafka


Description
---

Patch for KAFKA-1986


Diffs
-

  core/src/main/scala/kafka/server/ReplicaManager.scala 
586cf4caa95f58d5b2e6c7429732d25d2b3635c8 

Diff: https://reviews.apache.org/r/31449/diff/


Testing
---


Thanks,

Aditya Auradkar

[jira] [Created] (KAFKA-1986) Producer request failure rate should not include InvalidMessageSizeException and OffsetOutOfRangeException

[Aditya Auradkar (JIRA)]

Aditya Auradkar created KAFKA-1986:
--

 Summary: Producer request failure rate should not include 
InvalidMessageSizeException and OffsetOutOfRangeException
 Key: KAFKA-1986
 URL: https://issues.apache.org/jira/browse/KAFKA-1986
 Project: Kafka
  Issue Type: Sub-task
Reporter: Aditya Auradkar
Assignee: Aditya Auradkar


InvalidMessageSizeException and OffsetOutOfRangeException should not be counted 
a failedProduce and failedFetch requests since they are client side errors. 
They should be treated similarly to UnknownTopicOrPartitionException or 
NotLeaderForPartitionException



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

[Bhavesh Mistry]

Hi Jiangjie,

It might be too late.  But, I wanted to bring-up following use case for
adopting new MM:

1) Ability to publish message from src topic to different destination topic
via --overidenTopics=srcTopic:newDestinationTopic

In order to adopt, new MM enhancement customer will compare performance of
new MM and data quality while running  old MM against same destination
cluster in Prod.

Let me know if you agree to that or not.  Also, If yes, will be able to
able to provide this feature in release version.

Thanks,

Bhavesh


On Tue, Feb 24, 2015 at 5:31 PM, Jiangjie Qin 
wrote:

> Sure! Just created the voting thread :)
>
> On 2/24/15, 4:44 PM, "Jay Kreps"  wrote:
>
> >Hey Jiangjie,
> >
> >Let's do an official vote so that we know what we are voting on and we are
> >crisp on what the outcome was. This thread is very long :-
> >
> >-Jay
> >
> >On Tue, Feb 24, 2015 at 2:53 PM, Jiangjie Qin 
> >wrote:
> >
> >> I updated the KIP page based on the discussion we had.
> >>
> >> Should I launch another vote or we can think of this mail thread has
> >> already included a vote?
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On 2/11/15, 5:15 PM, "Neha Nakhede"  wrote:
> >>
> >> >Thanks for the explanation, Joel! Would love to see the results of the
> >> >throughput experiment and I'm a +1 on everything else, ncluding the
> >> >rebalance callback and record handler.
> >> >
> >> >-Neha
> >> >
> >> >On Wed, Feb 11, 2015 at 1:13 PM Jay Kreps  wrote:
> >> >
> >> >> Cool, I agree with all that.
> >> >>
> >> >> I agree about the need for a rebalancing callback.
> >> >>
> >> >> Totally agree about record handler.
> >> >>
> >> >> It would be great to see if a prototype of this is workable.
> >> >>
> >> >> Thanks guys!
> >> >>
> >> >> -Jay
> >> >>
> >> >> On Wed, Feb 11, 2015 at 12:36 PM, Joel Koshy 
> >> >>wrote:
> >> >>
> >> >> > Hey Jay,
> >> >> >
> >> >> > Guozhang, Becket and I got together to discuss this and we think:
> >> >> >
> >> >> > - It seems that your proposal based on the new consumr and flush
> >>call
> >> >> >   should work.
> >> >> > - We would likely need to call the poll with a timeout that matches
> >> >> >   the offset commit interval in order to deal with low volume
> >> >> >   mirroring pipelines.
> >> >> > - We will still need a rebalance callback to reduce duplicates -
> >>the
> >> >> >   rebalance callback would need to flush and commit offsets.
> >> >> > - The only remaining question is if the overall throughput is
> >> >> >   sufficient. I think someone at LinkedIn (I don't remember who)
> >>did
> >> >> >   some experiments with data channel size == 1 and ran into issues.
> >> >> >   That was not thoroughly investigated though.
> >> >> > - The addition of flush may actually make this solution viable for
> >>the
> >> >> >   current mirror-maker (wih the old consumer). We can prototype
> >>that
> >> >> >   offline and if it works out well we can redo KAFKA-1650 (i.e.,
> >> >> >   refactor the current mirror maker). The flush call and the new
> >> >> >   consumer didn't exist at the time we did KAFKA-1650 so this did
> >>not
> >> >> >   occur to us.
> >> >> > - We think the RecordHandler is still a useful small addition for
> >>the
> >> >> >   use-cases mentioned earlier in this thread.
> >> >> >
> >> >> > Thanks,
> >> >> >
> >> >> > Joel
> >> >> >
> >> >> > On Wed, Feb 11, 2015 at 09:05:39AM -0800, Jay Kreps wrote:
> >> >> > > Guozhang, I agree with 1-3, I do think what I was proposing was
> >> >>simpler
> >> >> > but
> >> >> > > perhaps there re gaps in that?
> >> >> > >
> >> >> > > Hey Joel--Here was a sketch of what I was proposing. I do think
> >>this
> >> >> > get's
> >> >> > > rid of manual offset tracking, especially doing so across threads
> >> >>with
> >> >> > > dedicated commit threads, which I think is prety complex.
> >> >> > >
> >> >> > > while(true) {
> >> >> > > val recs = consumer.poll(Long.MaxValue);
> >> >> > > for (rec <- recs)
> >> >> > > producer.send(rec, logErrorCallback)
> >> >> > > if(System.currentTimeMillis - lastCommit > commitInterval) {
> >> >> > > producer.flush()
> >> >> > > consumer.commit()
> >> >> > > lastCommit = System.currentTimeMillis
> >> >> > > }
> >> >> > > }
> >> >> > >
> >> >> > > (See the previous email for details). I think the question is: is
> >> >>there
> >> >> > any
> >> >> > > reason--performance, correctness, etc--that this won't work?
> >> >>Basically
> >> >> I
> >> >> > > think you guys have thought about this more so I may be missing
> > >> > something.
> >> >> > > If so let's flag it while we still have leeway on the consumer.
> >> >> > >
> >> >> > > If we think that will work, well I do think it is conceptually a
> >>lot
> >> >> > > simpler than the current code, though I suppose one could
> >>disagree
> >> >>on
> >> >> > that.
> >> >> > >
> >> >> > > -Jay
> >> >> > >
> >> >> > > On Wed, Feb 11, 2015 at 5:53 AM, Joel Koshy  >
> >> >> wrote:
> >> >> > >
> >> >> > > > Hi Jay,
> >> >> > > >
> >> >> >

[jira] [Commented] (KAFKA-1881) transient unit test failure in testDeleteTopicWithCleaner due to OOME

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337455#comment-14337455
 ] 

Ewen Cheslack-Postava commented on KAFKA-1881:
--

Created reviewboard https://reviews.apache.org/r/31447/diff/
 against branch origin/trunk

> transient unit test failure in testDeleteTopicWithCleaner due to OOME
> -
>
> Key: KAFKA-1881
> URL: https://issues.apache.org/jira/browse/KAFKA-1881
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.3
>Reporter: Jun Rao
>Assignee: Gwen Shapira
> Attachments: KAFKA-1881.patch
>
>
> kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner FAILED
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:42)
> at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:177)
> at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:86)
> at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:86)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.log.LogCleaner.(LogCleaner.scala:86)
> at kafka.log.LogManager.(LogManager.scala:64)
> at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:337)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
> at kafka.utils.TestUtils$.createServer(TestUtils.scala:134)
> at 
> kafka.admin.DeleteTopicTest$$anonfun$10.apply(DeleteTopicTest.scala:272)
> at 
> kafka.admin.DeleteTopicTest$$anonfun$10.apply(DeleteTopicTest.scala:272)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 
> kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:272)
> at 
> kafka.admin.DeleteTopicTest.testDeleteTopicWithCleaner(DeleteTopicTest.scala:241)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1881) transient unit test failure in testDeleteTopicWithCleaner due to OOME

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-1881:
-
Attachment: KAFKA-1881.patch

> transient unit test failure in testDeleteTopicWithCleaner due to OOME
> -
>
> Key: KAFKA-1881
> URL: https://issues.apache.org/jira/browse/KAFKA-1881
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.3
>Reporter: Jun Rao
>Assignee: Gwen Shapira
> Attachments: KAFKA-1881.patch
>
>
> kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner FAILED
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:42)
> at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:177)
> at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:86)
> at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:86)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.log.LogCleaner.(LogCleaner.scala:86)
> at kafka.log.LogManager.(LogManager.scala:64)
> at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:337)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
> at kafka.utils.TestUtils$.createServer(TestUtils.scala:134)
> at 
> kafka.admin.DeleteTopicTest$$anonfun$10.apply(DeleteTopicTest.scala:272)
> at 
> kafka.admin.DeleteTopicTest$$anonfun$10.apply(DeleteTopicTest.scala:272)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 
> kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:272)
> at 
> kafka.admin.DeleteTopicTest.testDeleteTopicWithCleaner(DeleteTopicTest.scala:241)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1881) transient unit test failure in testDeleteTopicWithCleaner due to OOME

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-1881:
-
Assignee: Ewen Cheslack-Postava  (was: Gwen Shapira)
  Status: Patch Available  (was: Open)

> transient unit test failure in testDeleteTopicWithCleaner due to OOME
> -
>
> Key: KAFKA-1881
> URL: https://issues.apache.org/jira/browse/KAFKA-1881
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.3
>Reporter: Jun Rao
>Assignee: Ewen Cheslack-Postava
> Attachments: KAFKA-1881.patch
>
>
> kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner FAILED
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:42)
> at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:177)
> at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:86)
> at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:86)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.log.LogCleaner.(LogCleaner.scala:86)
> at kafka.log.LogManager.(LogManager.scala:64)
> at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:337)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
> at kafka.utils.TestUtils$.createServer(TestUtils.scala:134)
> at 
> kafka.admin.DeleteTopicTest$$anonfun$10.apply(DeleteTopicTest.scala:272)
> at 
> kafka.admin.DeleteTopicTest$$anonfun$10.apply(DeleteTopicTest.scala:272)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 
> kafka.admin.DeleteTopicTest.createTestTopicAndCluster(DeleteTopicTest.scala:272)
> at 
> kafka.admin.DeleteTopicTest.testDeleteTopicWithCleaner(DeleteTopicTest.scala:241)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1400) transient unit test failure in SocketServerTest

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337452#comment-14337452
 ] 

Gwen Shapira commented on KAFKA-1400:
-

 and here are the logs :

{code}
[2015-02-25 15:11:10,002] INFO Awaiting socket connections on 0.0.0.0:52503. 
(kafka.network.Acceptor:68)
[2015-02-25 15:11:10,007] INFO [Socket Server on Broker 0], Started 
(kafka.network.SocketServer:68)
[2015-02-25 15:11:10,096] INFO Awaiting socket connections on 0.0.0.0:52504. 
(kafka.network.Acceptor:68)
[2015-02-25 15:11:10,096] INFO [Socket Server on Broker 0], Started 
(kafka.network.SocketServer:68)
wrote data to socket Socket[addr=localhost/127.0.0.1,port=52504,localport=52505]
[2015-02-25 15:11:10,111] DEBUG Accepted connection from /127.0.0.1 on 
/127.0.0.1:52504. sendBufferSize [actual|requested]: [30|30] 
recvBufferSize [actual|requested]: [310308|30] (kafka.network.Acceptor:52)
[2015-02-25 15:11:10,114] TRACE Processor id 0 selection time = 17218000 ns 
(kafka.network.Processor:36)
[2015-02-25 15:11:10,115] DEBUG Processor 0 listening to new connection from 
/127.0.0.1:52505 (kafka.network.Processor:52)
[2015-02-25 15:11:10,115] TRACE Processor id 0 selection time = 36000 ns 
(kafka.network.Processor:36)
[2015-02-25 15:11:10,121] TRACE 42 bytes read from /127.0.0.1:52505 
(kafka.network.Processor:36)
[2015-02-25 15:11:10,160] TRACE Processor 1 received request : Name: 
ProducerRequest; Version: 0; CorrelationId: 0; ClientId: ; RequiredAcks: 0; 
AckTimeoutMs: 0 ms; TopicAndPartition:  (kafka.network.RequestChannel$:36)
[2015-02-25 15:11:10,161] TRACE Processor 0 received request : Name: 
ProducerRequest; Version: 0; CorrelationId: 0; ClientId: ; RequiredAcks: 0; 
AckTimeoutMs: 0 ms; TopicAndPartition:  (kafka.network.RequestChannel$:36)
[2015-02-25 15:11:10,165] TRACE Processor id 0 selection time = 3599000 ns 
(kafka.network.Processor:36)
[2015-02-25 15:11:10,165] INFO [Socket Server on Broker 0], Shutting down 
(kafka.network.SocketServer:68)
[2015-02-25 15:11:10,166] DEBUG Closing server socket and selector. 
(kafka.network.Acceptor:52)
[2015-02-25 15:11:10,166] TRACE Socket server received response to send, 
registering for write: 
Response(0,Request(0,sun.nio.ch.SelectionKeyImpl@4328493c,null,1424905870123,/127.0.0.1:52505),kafka.network.BoundedByteBufferSend@318b1420,SendAction)
 (kafka.network.Processor:36)
[2015-02-25 15:11:10,167] TRACE Processor id 0 selection time = 26000 ns 
(kafka.network.Processor:36)
[2015-02-25 15:11:10,170] TRACE 22 bytes written to /127.0.0.1:52505 using key 
sun.nio.ch.SelectionKeyImpl@4328493c (kafka.network.Processor:36)
[2015-02-25 15:11:10,170] DEBUG is socket 
ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=52504] closed? true 
(kafka.network.Acceptor:52)
[2015-02-25 15:11:10,171] DEBUG Done shutting down acceptor. 
(kafka.network.Acceptor:52)
[2015-02-25 15:11:10,231] TRACE Completed request:Name: ProducerRequest; 
Version: 0; CorrelationId: 0; ClientId: ; RequiredAcks: 0; AckTimeoutMs: 0 ms; 
TopicAndPartition:  from client 
/127.0.0.1:52505;totalTime:47,requestQueueTime:0,localTime:1424905870165,remoteTime:0,responseQueueTime:1,sendTime:5
 (kafka.request.logger:85)
[2015-02-25 15:11:10,232] TRACE Finished writing, registering for read on 
connection /127.0.0.1:52505 (kafka.network.Processor:36)
[2015-02-25 15:11:10,232] DEBUG Closing selector. (kafka.network.Processor:52)
[2015-02-25 15:11:10,233] DEBUG Closing connection from /127.0.0.1:52505 
(kafka.network.Processor:52)
[2015-02-25 15:11:10,237] DEBUG done shutting down processor 
(kafka.network.Processor:52)
[2015-02-25 15:11:10,237] INFO [Socket Server on Broker 0], SocketServer: 
Shutdown completed (kafka.network.SocketServer:68)
wrote data to socket Socket[addr=localhost/127.0.0.1,port=52504,localport=52505]
- testSocketsCloseOnShutdown
{code}

Note that we successfully wrote to the socket (... the "wrote data" line is 
logged after calling flush() and sending the request). This is with trunk code 
(I added few extra log lines for clarity).

Are we sure that writing a single packet (we are not sending a lot of data) to 
a server that did socket.close() is actually expected to fail?

Because it looks like this may not be the case:
http://stackoverflow.com/questions/11436013/writing-to-a-closed-local-tcp-socket-not-failing


> transient unit test failure in SocketServerTest
> ---
>
> Key: KAFKA-1400
> URL: https://issues.apache.org/jira/browse/KAFKA-1400
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
>Reporter: Jun Rao
>Assignee: Neha Narkhede
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-1400.patch
>
>
> Saw the following transient failure.
> kafka.network.SocketServerTest > testSocketsCloseOnShutdown FAILED 
> java.la

Review Request 31447: Patch for KAFKA-1881

[Ewen Cheslack-Postava]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31447/
---

Review request for kafka.


Bugs: KAFKA-1881
https://issues.apache.org/jira/browse/KAFKA-1881


Repository: kafka


Description
---

KAFKA-1881: Fix out of memory error in testDeleteTopicWithCleaner by making the 
dedupe buffer much smaller.


Diffs
-

  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
0cbd72684a0cc6fe9f1fba2dc834de675b2eb994 

Diff: https://reviews.apache.org/r/31447/diff/


Testing
---


Thanks,

Ewen Cheslack-Postava

[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

[Jonathan Creasy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337379#comment-14337379
 ] 

Jonathan Creasy commented on KAFKA-1664:


I think this is because it expects the string to be:

baelish-001.edh.cloudera.com:2181,baelish-002.edh.cloudera.com:2181,baelis
h-003.edh.cloudera.com:2181/kafka


I don¹t think you would ever want to have a different root per host.

> Kafka does not properly parse multiple ZK nodes with non-root chroot
> 
>
> Key: KAFKA-1664
> URL: https://issues.apache.org/jira/browse/KAFKA-1664
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Ricky Saltzer
>Assignee: Ashish K Singh
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-1664.1.patch, KAFKA-1664.2.patch, 
> KAFKA-1664.patch, KAFKA-1664_2015-01-29_10:26:20.patch, 
> KAFKA-1664_2015-02-24_11:02:23.patch
>
>
> When using a non-root ZK directory for Kafka, if you specify multiple ZK 
> servers, Kafka does not seem to properly parse the connection string. 
> *Error*
> {code}
> [root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper 
> baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181/kafka,baelish-003.edh.cloudera.com:2181/kafka
>  --topic test-topic
> [2014-10-01 15:31:04,629] ERROR Error processing message, stopping consumer:  
> (kafka.consumer.ConsoleConsumer$)
> java.lang.IllegalArgumentException: Path length must be > 0
>   at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
>   at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
>   at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
>   at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
>   at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
>   at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
>   at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
>   at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>   at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
>   at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
>   at 
> kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
>   at 
> kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils.scala:306)
>   at 
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:755)
>   at 
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
>   at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
>   at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
> {code}
> *Working*
> {code}
> [root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper 
> baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

[Jonathan Creasy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337382#comment-14337382
 ] 

Jonathan Creasy commented on KAFKA-1664:


nevermind, missed Jun's response farther up. Carry on! :)

> Kafka does not properly parse multiple ZK nodes with non-root chroot
> 
>
> Key: KAFKA-1664
> URL: https://issues.apache.org/jira/browse/KAFKA-1664
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Ricky Saltzer
>Assignee: Ashish K Singh
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-1664.1.patch, KAFKA-1664.2.patch, 
> KAFKA-1664.patch, KAFKA-1664_2015-01-29_10:26:20.patch, 
> KAFKA-1664_2015-02-24_11:02:23.patch
>
>
> When using a non-root ZK directory for Kafka, if you specify multiple ZK 
> servers, Kafka does not seem to properly parse the connection string. 
> *Error*
> {code}
> [root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper 
> baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181/kafka,baelish-003.edh.cloudera.com:2181/kafka
>  --topic test-topic
> [2014-10-01 15:31:04,629] ERROR Error processing message, stopping consumer:  
> (kafka.consumer.ConsoleConsumer$)
> java.lang.IllegalArgumentException: Path length must be > 0
>   at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
>   at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
>   at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
>   at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
>   at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
>   at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
>   at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
>   at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>   at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>   at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
>   at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
>   at 
> kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
>   at 
> kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils.scala:306)
>   at 
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scala:755)
>   at 
> kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:145)
>   at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
>   at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
> {code}
> *Working*
> {code}
> [root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper 
> baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

[Jonathan Creasy]

I think this is because it expects the string to be:

 baelish-001.edh.cloudera.com:2181,baelish-002.edh.cloudera.com:2181,baelis
h-003.edh.cloudera.com:2181/kafka


I don¹t think you would ever want to have a different root per host.

-Jonathan

On 2/24/15, 1:04 PM, "Ashish Singh (JIRA)"  wrote:

>
>[ 
>https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.p
>lugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335255#com
>ment-14335255 ] 
>
>Ashish Singh commented on KAFKA-1664:
>-
>
>[~nehanarkhede] addressed the warnings issue. Thanks for the review.
>
>> Kafka does not properly parse multiple ZK nodes with non-root chroot
>> 
>>
>> Key: KAFKA-1664
>> URL: https://issues.apache.org/jira/browse/KAFKA-1664
>> Project: Kafka
>>  Issue Type: Bug
>>  Components: clients
>>Reporter: Ricky Saltzer
>>Assignee: Ashish Singh
>>Priority: Minor
>>  Labels: newbie
>> Attachments: KAFKA-1664.1.patch, KAFKA-1664.2.patch,
>>KAFKA-1664.patch, KAFKA-1664_2015-01-29_10:26:20.patch,
>>KAFKA-1664_2015-02-24_11:02:23.patch
>>
>>
>> When using a non-root ZK directory for Kafka, if you specify multiple
>>ZK servers, Kafka does not seem to properly parse the connection string.
>> *Error*
>> {code}
>> [root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
>>baelish-001.edh.cloudera.com:2181/kafka,baelish-002.edh.cloudera.com:2181
>>/kafka,baelish-003.edh.cloudera.com:2181/kafka --topic test-topic
>> [2014-10-01 15:31:04,629] ERROR Error processing message, stopping
>>consumer:  (kafka.consumer.ConsoleConsumer$)
>> java.lang.IllegalArgumentException: Path length must be > 0
>>  at 
>>org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:48)
>>  at 
>>org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:35)
>>  at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:766)
>>  at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
>>  at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
>>  at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
>>  at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
>>  at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
>>  at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:213)
>>  at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>>  at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>>  at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:223)
>>  at kafka.utils.ZkUtils$.createParentPath(ZkUtils.scala:245)
>>  at kafka.utils.ZkUtils$.createEphemeralPath(ZkUtils.scala:256)
>>  at 
>>kafka.utils.ZkUtils$.createEphemeralPathExpectConflict(ZkUtils.scala:268)
>>  at 
>>kafka.utils.ZkUtils$.createEphemeralPathExpectConflictHandleZKBug(ZkUtils
>>.scala:306)
>>  at 
>>kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsume
>>rConnector$$registerConsumerInZK(ZookeeperConsumerConnector.scala:226)
>>  at 
>>kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(Z
>>ookeeperConsumerConnector.scala:755)
>>  at 
>>kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(Zo
>>okeeperConsumerConnector.scala:145)
>>  at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:196)
>>  at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
>> {code}
>> *Working*
>> {code}
>> [root@hodor-001 bin]# ./kafka-console-consumer.sh --zookeeper
>>baelish-001.edh.cloudera.com:2181/kafka --topic test-topic
>> {code}
>
>
>
>--
>This message was sent by Atlassian JIRA
>(v6.3.4#6332)

[jira] [Commented] (KAFKA-1985) Document on possible error codes for each response type

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337214#comment-14337214
 ] 

Jay Kreps commented on KAFKA-1985:
--

Totally agree. The documentation for this should really go with the protocol 
description, though. All the other clients have the same issue we do only much 
worse because they may be less fluent in Java.

> Document on possible error codes for each response type
> ---
>
> Key: KAFKA-1985
> URL: https://issues.apache.org/jira/browse/KAFKA-1985
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Guozhang Wang
> Fix For: 0.9.0
>
>
> When coding the clients logic we tend to forget one or more possible error 
> codes that needs special handling because it is not summarized and documented 
> anywhere. It would better to at least add comments in
> {code}
> org.apache.kafka.common.requests.XXResponse
> {code}
> about all possible error codes so that people can check and handle them 
> appropriately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1985) Document on possible error codes for each response type

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337193#comment-14337193
 ] 

Gwen Shapira commented on KAFKA-1985:
-

Awesome awesome idea.

However, we should also document that this is not a guarantee - new error codes 
may appear in new releases with or without a protocol bump and clients must be 
able to handle unknown error codes gracefully.

> Document on possible error codes for each response type
> ---
>
> Key: KAFKA-1985
> URL: https://issues.apache.org/jira/browse/KAFKA-1985
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Guozhang Wang
> Fix For: 0.9.0
>
>
> When coding the clients logic we tend to forget one or more possible error 
> codes that needs special handling because it is not summarized and documented 
> anywhere. It would better to at least add comments in
> {code}
> org.apache.kafka.common.requests.XXResponse
> {code}
> about all possible error codes so that people can check and handle them 
> appropriately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: KafkaPreCommit #20

[Apache Jenkins Server]

See 

Changes:

[jjkoshy] KAFKA-1953; KAFKA-1962; Disambiguate purgatory metrics; restore 
delayed request metrics; reviewed by Guozhang Wang

--
[...truncated 834 lines...]
kafka.admin.AdminTest > testShutdownBroker PASSED

kafka.admin.AdminTest > testTopicConfigChange PASSED

kafka.admin.DeleteConsumerGroupTest > testGroupWideDeleteInZK PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKForGroupConsumingOneTopic PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKForGroupConsumingMultipleTopics PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKDoesNothingForActiveGroupConsumingMultipleTopics 
PASSED

kafka.admin.DeleteConsumerGroupTest > testTopicWideDeleteInZK PASSED

kafka.admin.DeleteConsumerGroupTest > 
testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.RollingBounceTest > testRollingBounce PASSED

kafka.integration.TopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testOffsetAssignment PASSED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testEquality PASSED

kafka.message.MessageTest > testIsHashable PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndHoursProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testAdvertiseDefaults PASSED

kafka.server.KafkaConfigTest > testAdvertiseConfigured PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testUncleanElectionDisabled PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeBothMsAndHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testDefaultCompressionType PA

RE: Announcing the Confluent Platform built on Apache Kafka

[Aditya Auradkar]

Congrats!


From: Andrew Otto [ao...@wikimedia.org]
Sent: Wednesday, February 25, 2015 12:06 PM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Announcing the Confluent Platform built on Apache Kafka

Wow, .deb packages.  I love you.


> On Feb 25, 2015, at 14:48, Joseph Lawson  wrote:
>
> This is really awesome stuff.  It's great to see y'all growing!  Thank you 
> and congratulations!
>
> 
> From: Neha Narkhede 
> Sent: Wednesday, February 25, 2015 1:31 PM
> To: us...@kafka.apache.org; dev@kafka.apache.org
> Subject: Announcing the Confluent Platform built on Apache Kafka
>
> Folks,
>
> We, at Confluent , are excited to announce the release
> of Confluent Platform 1.0 built around Apache Kafka -
> http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0/
>
> We also published a detailed two-part guide on how you can put Kafka to use
> in your organization -
> http://blog.confluent.io/2015/02/25/stream-data-platform-1/
>
> And, there is a public mailing list where we would love to hear your
> feedback: confluent-platf...@googlegroups.com
>
> Thanks,
> Neha

Unit tests in java7 vs java8

[Gwen Shapira]

Hi,

Anyone running tests on Java 8? I just noticed that they take almost twice
as long to run compared to Java 7 (at least on my box, and with Scala
2.10.4).

Anyone else noticed this? Maybe even did some digging on the causes?

Gwen

Build failed in Jenkins: Kafka-trunk #407

[Apache Jenkins Server]

See 

Changes:

[jjkoshy] KAFKA-1953; KAFKA-1962; Disambiguate purgatory metrics; restore 
delayed request metrics; reviewed by Guozhang Wang

--
[...truncated 820 lines...]
kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKDoesNothingForActiveGroupConsumingMultipleTopics 
PASSED

kafka.admin.DeleteConsumerGroupTest > testTopicWideDeleteInZK PASSED

kafka.admin.DeleteConsumerGroupTest > 
testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.RollingBounceTest > testRollingBounce PASSED

kafka.integration.TopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testOffsetAssignment PASSED

kafka.message.MessageTest > testFieldValues PASSED

kafka.message.MessageTest > testChecksum PASSED

kafka.message.MessageTest > testEquality PASSED

kafka.message.MessageTest > testIsHashable PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndHoursProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testAdvertiseDefaults PASSED

kafka.server.KafkaConfigTest > testAdvertiseConfigured PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testUncleanElectionDisabled PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeBothMsAndHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testDefaultCompressionType PASSED

kafka.server.KafkaConfigTest > testValidCompressionType PASSED

kafka.server.KafkaConfigTest > testInvalidCompressionType PASSED

kafka.server.DynamicConfigChangeTest > testConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic PASSED

kafka.server.LogOffsetTest > testGetOffsetsForUnknownTopic PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeLatestTime PASSED

kafka.server.LogOffsetTest > testEmptyLogsGetOffsets PASSED

kafka.serve

Re: Announcing the Confluent Platform built on Apache Kafka

[Andrew Otto]

Wow, .deb packages.  I love you.


> On Feb 25, 2015, at 14:48, Joseph Lawson  wrote:
> 
> This is really awesome stuff.  It's great to see y'all growing!  Thank you 
> and congratulations!
> 
> 
> From: Neha Narkhede 
> Sent: Wednesday, February 25, 2015 1:31 PM
> To: us...@kafka.apache.org; dev@kafka.apache.org
> Subject: Announcing the Confluent Platform built on Apache Kafka
> 
> Folks,
> 
> We, at Confluent , are excited to announce the release
> of Confluent Platform 1.0 built around Apache Kafka -
> http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0/
> 
> We also published a detailed two-part guide on how you can put Kafka to use
> in your organization -
> http://blog.confluent.io/2015/02/25/stream-data-platform-1/
> 
> And, there is a public mailing list where we would love to hear your
> feedback: confluent-platf...@googlegroups.com
> 
> Thanks,
> Neha

Re: Announcing the Confluent Platform built on Apache Kafka

[Ashish Singh]

Congrats guys!

On Wed, Feb 25, 2015 at 11:47 AM, Jiangjie Qin 
wrote:

> Congrats!
>
> On 2/25/15, 11:30 AM, "Guozhang Wang"  wrote:
>
> >+1, very nice!
> >
> >On Wed, Feb 25, 2015 at 11:20 AM, Sriram Subramanian <
> >srsubraman...@linkedin.com.invalid> wrote:
> >
> >> Congratulations!
> >>
> >> On 2/25/15 11:15 AM, "Joe Stein"  wrote:
> >>
> >> >Awesome!
> >> >
> >> >The future of schema management, has arrived
> >> >
> >> >~ Joestein
> >> >
> >> >On Wed, Feb 25, 2015 at 1:34 PM, Gwen Shapira 
> >> >wrote:
> >> >
> >> >> Congrats, Confluent team.
> >> >>
> >> >> This is super exciting :)
> >> >>
> >> >>
> >> >> On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede 
> >> >>wrote:
> >> >>
> >> >> > Folks,
> >> >> >
> >> >> > We, at Confluent , are excited to announce
> the
> >> >> > release
> >> >> > of Confluent Platform 1.0 built around Apache Kafka -
> >> >> >
> >> >>
> >> >>
> >>
> >>
> http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0
> >> >>/
> >> >> >
> >> >> > We also published a detailed two-part guide on how you can put
> >>Kafka
> >> >>to
> >> >> use
> >> >> > in your organization -
> >> >> > http://blog.confluent.io/2015/02/25/stream-data-platform-1/
> >> >> >
> >> >> > And, there is a public mailing list where we would love to hear
> >>your
> >> >> > feedback: confluent-platf...@googlegroups.com
> >> >> >
> >> >> > Thanks,
> >> >> > Neha
> >> >> >
> >> >>
> >>
> >>
> >
> >
> >--
> >-- Guozhang
>
>


-- 

Regards,
Ashish

Tips for working with Kafka and data streams

[Jay Kreps]

Hey guys,

One thing we tried to do along with the product release was start to put
together a practical guide for using Kafka. I wrote this up here:
http://blog.confluent.io/2015/02/25/stream-data-platform-1/

I'd like to keep expanding on this as good practices emerge and we learn
more stuff. So two questions:
1. Anything you think other people should know about working with data
streams? What did you wish you knew when you got started?
2. Anything you don't know about but would like to hear more about?

-Jay

Re: Announcing the Confluent Platform built on Apache Kafka

[Jiangjie Qin]

Congrats!

On 2/25/15, 11:30 AM, "Guozhang Wang"  wrote:

>+1, very nice!
>
>On Wed, Feb 25, 2015 at 11:20 AM, Sriram Subramanian <
>srsubraman...@linkedin.com.invalid> wrote:
>
>> Congratulations!
>>
>> On 2/25/15 11:15 AM, "Joe Stein"  wrote:
>>
>> >Awesome!
>> >
>> >The future of schema management, has arrived
>> >
>> >~ Joestein
>> >
>> >On Wed, Feb 25, 2015 at 1:34 PM, Gwen Shapira 
>> >wrote:
>> >
>> >> Congrats, Confluent team.
>> >>
>> >> This is super exciting :)
>> >>
>> >>
>> >> On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede 
>> >>wrote:
>> >>
>> >> > Folks,
>> >> >
>> >> > We, at Confluent , are excited to announce the
>> >> > release
>> >> > of Confluent Platform 1.0 built around Apache Kafka -
>> >> >
>> >>
>> >>
>> 
>>http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0
>> >>/
>> >> >
>> >> > We also published a detailed two-part guide on how you can put
>>Kafka
>> >>to
>> >> use
>> >> > in your organization -
>> >> > http://blog.confluent.io/2015/02/25/stream-data-platform-1/
>> >> >
>> >> > And, there is a public mailing list where we would love to hear
>>your
>> >> > feedback: confluent-platf...@googlegroups.com
>> >> >
>> >> > Thanks,
>> >> > Neha
>> >> >
>> >>
>>
>>
>
>
>-- 
>-- Guozhang

Re: Announcing the Confluent Platform built on Apache Kafka

[Guozhang Wang]

+1, very nice!

On Wed, Feb 25, 2015 at 11:20 AM, Sriram Subramanian <
srsubraman...@linkedin.com.invalid> wrote:

> Congratulations!
>
> On 2/25/15 11:15 AM, "Joe Stein"  wrote:
>
> >Awesome!
> >
> >The future of schema management, has arrived
> >
> >~ Joestein
> >
> >On Wed, Feb 25, 2015 at 1:34 PM, Gwen Shapira 
> >wrote:
> >
> >> Congrats, Confluent team.
> >>
> >> This is super exciting :)
> >>
> >>
> >> On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede 
> >>wrote:
> >>
> >> > Folks,
> >> >
> >> > We, at Confluent , are excited to announce the
> >> > release
> >> > of Confluent Platform 1.0 built around Apache Kafka -
> >> >
> >>
> >>
> http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0
> >>/
> >> >
> >> > We also published a detailed two-part guide on how you can put Kafka
> >>to
> >> use
> >> > in your organization -
> >> > http://blog.confluent.io/2015/02/25/stream-data-platform-1/
> >> >
> >> > And, there is a public mailing list where we would love to hear your
> >> > feedback: confluent-platf...@googlegroups.com
> >> >
> >> > Thanks,
> >> > Neha
> >> >
> >>
>
>


-- 
-- Guozhang

Re: Announcing the Confluent Platform built on Apache Kafka

[Harsha]

Congrats!
-Harsha

On Wed, Feb 25, 2015, at 11:20 AM, Sriram Subramanian wrote:
> Congratulations! 
> 
> On 2/25/15 11:15 AM, "Joe Stein"  wrote:
> 
> >Awesome!
> >
> >The future of schema management, has arrived
> >
> >~ Joestein
> >
> >On Wed, Feb 25, 2015 at 1:34 PM, Gwen Shapira 
> >wrote:
> >
> >> Congrats, Confluent team.
> >>
> >> This is super exciting :)
> >>
> >>
> >> On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede 
> >>wrote:
> >>
> >> > Folks,
> >> >
> >> > We, at Confluent , are excited to announce the
> >> > release
> >> > of Confluent Platform 1.0 built around Apache Kafka -
> >> >
> >> 
> >>http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0
> >>/
> >> >
> >> > We also published a detailed two-part guide on how you can put Kafka
> >>to
> >> use
> >> > in your organization -
> >> > http://blog.confluent.io/2015/02/25/stream-data-platform-1/
> >> >
> >> > And, there is a public mailing list where we would love to hear your
> >> > feedback: confluent-platf...@googlegroups.com
> >> >
> >> > Thanks,
> >> > Neha
> >> >
> >>
> 

Re: Announcing the Confluent Platform built on Apache Kafka

[Sriram Subramanian]

Congratulations! 

On 2/25/15 11:15 AM, "Joe Stein"  wrote:

>Awesome!
>
>The future of schema management, has arrived
>
>~ Joestein
>
>On Wed, Feb 25, 2015 at 1:34 PM, Gwen Shapira 
>wrote:
>
>> Congrats, Confluent team.
>>
>> This is super exciting :)
>>
>>
>> On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede 
>>wrote:
>>
>> > Folks,
>> >
>> > We, at Confluent , are excited to announce the
>> > release
>> > of Confluent Platform 1.0 built around Apache Kafka -
>> >
>> 
>>http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0
>>/
>> >
>> > We also published a detailed two-part guide on how you can put Kafka
>>to
>> use
>> > in your organization -
>> > http://blog.confluent.io/2015/02/25/stream-data-platform-1/
>> >
>> > And, there is a public mailing list where we would love to hear your
>> > feedback: confluent-platf...@googlegroups.com
>> >
>> > Thanks,
>> > Neha
>> >
>>

Re: Announcing the Confluent Platform built on Apache Kafka

[Joe Stein]

Awesome!

The future of schema management, has arrived

~ Joestein

On Wed, Feb 25, 2015 at 1:34 PM, Gwen Shapira  wrote:

> Congrats, Confluent team.
>
> This is super exciting :)
>
>
> On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede  wrote:
>
> > Folks,
> >
> > We, at Confluent , are excited to announce the
> > release
> > of Confluent Platform 1.0 built around Apache Kafka -
> >
> http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0/
> >
> > We also published a detailed two-part guide on how you can put Kafka to
> use
> > in your organization -
> > http://blog.confluent.io/2015/02/25/stream-data-platform-1/
> >
> > And, there is a public mailing list where we would love to hear your
> > feedback: confluent-platf...@googlegroups.com
> >
> > Thanks,
> > Neha
> >
>

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336985#comment-14336985
 ] 

Guozhang Wang commented on KAFKA-1910:
--

The uploaded patch contains multiple fixes to the related JIRAs as well as 
refactoring the new consumer itself. I will summarize them here instead of in 
the RB:

1. Fix ConsumerTest.testXXXwithBrokerFailure: in RestartDeadBroker we need to 
call startup() on the old brokers instead of creating new ones as the last 
approach will case the metadata to be mess up and cause the test to hang 
(KAFKA-1948). Also make sure the "test" topic is created with correct 
replication factor to avoid hanging when the only replica broker was shutdown.

2. Fix ConsumerTest's __consumer_offsets topic: when we call partitionFor() the 
__consumer_offsets topic may be created with replication as 
min(offsetTopicRaplicationFactor, aliveBrokers.size), see KAFKA-1864 for 
details (KAFKA-1975). 

3. Add the IllegalGeneration logic in the coordinator as it is important for 
consumers rebalancing after rediscovering the coordinator, in the current stub 
it always return OK and hence consumers migrating to the new coordinator will 
not trigger rebalance (KAFKA-1964).

4. Create the Coodinator and the FetchManager modules as KafkaConsumer 
internals. Coordinator is responsible for assign partitions (join groups), 
commit offsets and fetch offsets from coordinator, and FetchManager is 
responsible for handling fetch request / responses.

4.1 After the refactoring it is easier to detect and fix a bug where response 
callbacks being triggered multiple times, causing the coordinator NPE 
(KAFKA-1969).

4.2 Avoid always trying to fetch offsets from coordinator whenever the consumer 
decides to update fetch positions, introduce a few new variables / APIs in 
SubscriptionState accordingly.

4.3 Move serializer / de-serializer configs / constructors to AbstractConfig.

4.4 Add missing error handling in commit offset / heartbeat responses. In 
general I think we should make notes about possible error codes in each of the 
response type to help coding error handling logic, has filed KAFKA-1985 for 
that.

> Refactor KafkaConsumer
> --
>
> Key: KAFKA-1910
> URL: https://issues.apache.org/jira/browse/KAFKA-1910
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> KafkaConsumer now contains all the logic on the consumer side, making it a 
> very huge class file, better re-factoring it to have multiple layers on top 
> of KafkaClient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-1985) Document on possible error codes for each response type

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-1985:


 Summary: Document on possible error codes for each response type
 Key: KAFKA-1985
 URL: https://issues.apache.org/jira/browse/KAFKA-1985
 Project: Kafka
  Issue Type: Bug
  Components: clients
Reporter: Guozhang Wang
 Fix For: 0.9.0


When coding the clients logic we tend to forget one or more possible error 
codes that needs special handling because it is not summarized and documented 
anywhere. It would better to at least add comments in

{code}
org.apache.kafka.common.requests.XXResponse
{code}

about all possible error codes so that people can check and handle them 
appropriately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Announcing the Confluent Platform built on Apache Kafka

[Gwen Shapira]

Congrats, Confluent team.

This is super exciting :)


On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede  wrote:

> Folks,
>
> We, at Confluent , are excited to announce the
> release
> of Confluent Platform 1.0 built around Apache Kafka -
> http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0/
>
> We also published a detailed two-part guide on how you can put Kafka to use
> in your organization -
> http://blog.confluent.io/2015/02/25/stream-data-platform-1/
>
> And, there is a public mailing list where we would love to hear your
> feedback: confluent-platf...@googlegroups.com
>
> Thanks,
> Neha
>

Announcing the Confluent Platform built on Apache Kafka

[Neha Narkhede]

Folks,

We, at Confluent , are excited to announce the release
of Confluent Platform 1.0 built around Apache Kafka -
http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0/

We also published a detailed two-part guide on how you can put Kafka to use
in your organization -
http://blog.confluent.io/2015/02/25/stream-data-platform-1/

And, there is a public mailing list where we would love to hear your
feedback: confluent-platf...@googlegroups.com

Thanks,
Neha

RE: Review Request 31385: Patch for KAFKA-1978

[Abhishek Nigam]

I added debug information and the reason is that the array size, 
replicaIdxMsgIdList is 2 but the number of files which are being validated are 
3.


2015-02-25 00:53:17,882 - INFO - array size: 2 (kafka_system_test_utils)
2015-02-25 00:53:17,882 - INFO - replicaFactor: 2 (kafka_system_test_utils)
2015-02-25 00:53:17,882 - INFO - replicaIdx: 3 (kafka_system_test_utils)

The code tries to index into the array using index 2 but since the size of the 
array is only 2 we get array out of bounds exception.

-Abhishek

From: Abhishek Nigam
Sent: Wednesday, February 25, 2015 9:43 AM
To: Guozhang Wang; kafka
Subject: RE: Review Request 31385: Patch for KAFKA-1978

2015-02-06 00:51:30,975 - INFO - 
==
2015-02-06 00:51:30,975 - INFO - Exception while running test list index out of 
range
2015-02-06 00:51:30,975 - INFO - 
==
Traceback (most recent call last):
  File 
"/mnt/u001/kafka_replication_system_test/system_test/replication_testsuite/replica_basic_test.py",
 line 434, in runTest

kafka_system_test_utils.validate_simple_consumer_data_matched_across_replicas(self.systemTestEnv,
 self.testcaseEnv)
  File 
"/mnt/u001/kafka_replication_system_test/system_test/utils/kafka_system_test_utils.py",
 line 2223, in validate_simple_consumer_data_matched_across_replicas
replicaIdxMsgIdList[replicaIdx - 1][topicPartition] = consumerMsgIdList
IndexError: list index out of range

-Abhishek

From: Guozhang Wang [nore...@reviews.apache.org] on behalf of Guozhang Wang 
[wangg...@gmail.com]
Sent: Tuesday, February 24, 2015 6:36 PM
To: Abhishek Nigam; Guozhang Wang; kafka
Subject: Re: Review Request 31385: Patch for KAFKA-1978

This is an automatically generated e-mail. To reply, visit: 
https://reviews.apache.org/r/31385/


>From the diff file itself it's a bit hard to understand the issue and the 
>solution as well. Could you elaborate on "an out of bounds exception due to 
>mis-configuration" on the ticket?


- Guozhang Wang


On February 24th, 2015, 11:31 p.m. UTC, Abhishek Nigam wrote:

Review request for kafka.
By Abhishek Nigam.

Updated Feb. 24, 2015, 11:31 p.m.

Bugs: KAFKA-1978
Repository: kafka
Description

Fixing configuration for testcase 0131


Diffs

  *   
system_test/replication_testsuite/testcase_0131/testcase_0131_properties.json 
(0324b6f327cb75389f9f851fa3ca744d22a5d915)

View Diff

RE: Review Request 31385: Patch for KAFKA-1978

[Abhishek Nigam]

2015-02-06 00:51:30,975 - INFO - 
==
2015-02-06 00:51:30,975 - INFO - Exception while running test list index out of 
range
2015-02-06 00:51:30,975 - INFO - 
==
Traceback (most recent call last):
  File 
"/mnt/u001/kafka_replication_system_test/system_test/replication_testsuite/replica_basic_test.py",
 line 434, in runTest

kafka_system_test_utils.validate_simple_consumer_data_matched_across_replicas(self.systemTestEnv,
 self.testcaseEnv)
  File 
"/mnt/u001/kafka_replication_system_test/system_test/utils/kafka_system_test_utils.py",
 line 2223, in validate_simple_consumer_data_matched_across_replicas
replicaIdxMsgIdList[replicaIdx - 1][topicPartition] = consumerMsgIdList
IndexError: list index out of range

-Abhishek

From: Guozhang Wang [nore...@reviews.apache.org] on behalf of Guozhang Wang 
[wangg...@gmail.com]
Sent: Tuesday, February 24, 2015 6:36 PM
To: Abhishek Nigam; Guozhang Wang; kafka
Subject: Re: Review Request 31385: Patch for KAFKA-1978

This is an automatically generated e-mail. To reply, visit: 
https://reviews.apache.org/r/31385/


>From the diff file itself it's a bit hard to understand the issue and the 
>solution as well. Could you elaborate on "an out of bounds exception due to 
>mis-configuration" on the ticket?


- Guozhang Wang


On February 24th, 2015, 11:31 p.m. UTC, Abhishek Nigam wrote:

Review request for kafka.
By Abhishek Nigam.

Updated Feb. 24, 2015, 11:31 p.m.

Bugs: KAFKA-1978
Repository: kafka
Description

Fixing configuration for testcase 0131


Diffs

  *   
system_test/replication_testsuite/testcase_0131/testcase_0131_properties.json 
(0324b6f327cb75389f9f851fa3ca744d22a5d915)

View Diff

[jira] [Commented] (KAFKA-1961) Looks like its possible to delete _consumer_offsets topic

[Ted Malaska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336675#comment-14336675
 ] 

Ted Malaska commented on KAFKA-1961:


Let me know if there are any other changes I need to make before this can get 
committed.

Thanks

> Looks like its possible to delete _consumer_offsets topic
> -
>
> Key: KAFKA-1961
> URL: https://issues.apache.org/jira/browse/KAFKA-1961
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>  Labels: newbie
> Attachments: KAFKA-1961.3.patch, KAFKA-1961.4.patch
>
>
> Noticed that kafka-topics.sh --delete can successfully delete internal topics 
> (__consumer_offsets).
> I'm pretty sure we want to prevent that, to avoid users shooting themselves 
> in the foot.
> Topic admin command should check for internal topics, just like 
> ReplicaManager does and not let users delete them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336656#comment-14336656
 ] 

Neha Narkhede commented on KAFKA-1792:
--

+1. To reduce back-n-forth, it is better to discuss and agree on the complete 
proposal on the KIP, before posting code. 

> change behavior of --generate to produce assignment config with fair replica 
> distribution and minimal number of reassignments
> -
>
> Key: KAFKA-1792
> URL: https://issues.apache.org/jira/browse/KAFKA-1792
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, 
> KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, 
> KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, 
> KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt, 
> rebalance_use_cases.txt
>
>
> Current implementation produces fair replica distribution between specified 
> list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth 
> broker id=3, 
> generate will create an assignment config which will redistribute replicas 
> fairly across brokers [0..3] 
> in the same way as those partitions were created from scratch. It will not 
> take into consideration current replica 
> assignment and accordingly will not try to minimize number of replica moves 
> between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved 
> --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas 
> assigned;
> - minimum of reassignments - number of replica moves between brokers will be 
> minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added 
> brokers 2 and 3):
> - broker - 0, 1, 2, 3 
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3 
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified 
> initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336573#comment-14336573
 ] 

Joe Stein commented on KAFKA-1792:
--

[~Dmitry Pekar] we should have this discussion on the KIP thread where we 
already had came up to discuss making backwards compatable == preserve 
functionality. So --generate would be for no changes is like it is and your new 
code would be --re-balance. We would deprecate --generate in the 0.9.0 release. 
Can you update the KIP and have the discussion continue on the mailing list 
thread please, thanks!

> change behavior of --generate to produce assignment config with fair replica 
> distribution and minimal number of reassignments
> -
>
> Key: KAFKA-1792
> URL: https://issues.apache.org/jira/browse/KAFKA-1792
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, 
> KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, 
> KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, 
> KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt, 
> rebalance_use_cases.txt
>
>
> Current implementation produces fair replica distribution between specified 
> list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth 
> broker id=3, 
> generate will create an assignment config which will redistribute replicas 
> fairly across brokers [0..3] 
> in the same way as those partitions were created from scratch. It will not 
> take into consideration current replica 
> assignment and accordingly will not try to minimize number of replica moves 
> between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved 
> --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas 
> assigned;
> - minimum of reassignments - number of replica moves between brokers will be 
> minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added 
> brokers 2 and 3):
> - broker - 0, 1, 2, 3 
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3 
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified 
> initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments

[Dmitry Pekar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336555#comment-14336555
 ] 

Dmitry Pekar commented on KAFKA-1792:
-

[~nehanarkhede]
the command is still called --generate. There is no --re-balance command.
The algorithm, invoked by --generate has changed, but the CLI itself is 
backward-compatible.

Please see my comments: 
1. KIP now has "Decommission broker command" section about 
"--decommission-broker" command.
Replaced references to --re-balance command, with --generate command to match 
the implementation.

2. Patch matches.

3.Updated ticket attachments to be actual:
- rebalance_use_cases.txt
- generate_alg_tests.txt

> change behavior of --generate to produce assignment config with fair replica 
> distribution and minimal number of reassignments
> -
>
> Key: KAFKA-1792
> URL: https://issues.apache.org/jira/browse/KAFKA-1792
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, 
> KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, 
> KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, 
> KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt, 
> rebalance_use_cases.txt
>
>
> Current implementation produces fair replica distribution between specified 
> list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth 
> broker id=3, 
> generate will create an assignment config which will redistribute replicas 
> fairly across brokers [0..3] 
> in the same way as those partitions were created from scratch. It will not 
> take into consideration current replica 
> assignment and accordingly will not try to minimize number of replica moves 
> between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved 
> --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas 
> assigned;
> - minimum of reassignments - number of replica moves between brokers will be 
> minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added 
> brokers 2 and 3):
> - broker - 0, 1, 2, 3 
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3 
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified 
> initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments

[Dmitry Pekar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitry Pekar updated KAFKA-1792:

Attachment: rebalance_use_cases.txt

> change behavior of --generate to produce assignment config with fair replica 
> distribution and minimal number of reassignments
> -
>
> Key: KAFKA-1792
> URL: https://issues.apache.org/jira/browse/KAFKA-1792
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, 
> KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, 
> KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, 
> KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt, 
> rebalance_use_cases.txt
>
>
> Current implementation produces fair replica distribution between specified 
> list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth 
> broker id=3, 
> generate will create an assignment config which will redistribute replicas 
> fairly across brokers [0..3] 
> in the same way as those partitions were created from scratch. It will not 
> take into consideration current replica 
> assignment and accordingly will not try to minimize number of replica moves 
> between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved 
> --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas 
> assigned;
> - minimum of reassignments - number of replica moves between brokers will be 
> minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added 
> brokers 2 and 3):
> - broker - 0, 1, 2, 3 
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3 
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified 
> initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments

[Dmitry Pekar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitry Pekar updated KAFKA-1792:

Attachment: (was: rebalance_use_cases.txt)

> change behavior of --generate to produce assignment config with fair replica 
> distribution and minimal number of reassignments
> -
>
> Key: KAFKA-1792
> URL: https://issues.apache.org/jira/browse/KAFKA-1792
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, 
> KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, 
> KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, 
> KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt
>
>
> Current implementation produces fair replica distribution between specified 
> list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth 
> broker id=3, 
> generate will create an assignment config which will redistribute replicas 
> fairly across brokers [0..3] 
> in the same way as those partitions were created from scratch. It will not 
> take into consideration current replica 
> assignment and accordingly will not try to minimize number of replica moves 
> between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved 
> --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas 
> assigned;
> - minimum of reassignments - number of replica moves between brokers will be 
> minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added 
> brokers 2 and 3):
> - broker - 0, 1, 2, 3 
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3 
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified 
> initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)