[jira] [Commented] (KAFKA-4430) Broker logging "Topic and partition to exceptions: [topic,6] -> kafka.common.MessageSizeTooLargeException"

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692402#comment-15692402
 ] 

huxi commented on KAFKA-4430:
-

Great find. And, why is the message compressed twice and how did you know it?

> Broker logging "Topic and partition to exceptions: [topic,6] -> 
> kafka.common.MessageSizeTooLargeException"
> --
>
> Key: KAFKA-4430
> URL: https://issues.apache.org/jira/browse/KAFKA-4430
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.1
> Environment: Production 
>Reporter: Srinivas Dhruvakumar
>  Labels: newbie
>
> I have a setup as below 
> DC Kafka 
> Mirrormaker 
> Aggregate Kafka
> Here is the following settings. I have set the max.message.bytes to 1M Bytes 
> on DC and AGG kafka side. Mirrormaker producer settings --  batch.size is set 
> to 500 K Bytes and max.request.size is set to 1 M Byte and ack to 0 , 
> compression-> gzip . 
> However on the Aggregate Kafka I get the following exception 
> Closing connection due to error during produce request with correlation id 
> 414156659 from client id producer-1 with ack=0
> Topic and partition to exceptions: [topic1,6] -> 
> kafka.common.MessageSizeTooLargeException
> Is this a bug or why would this happen. I have configured mirrormaker to send 
> messages less than 1 M Bytes . Are the messages getting dropped ? Under what 
> circumstances this error occurs ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #1059

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-4345) Run decktape test for each pull request

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692191#comment-15692191
 ] 

ASF GitHub Bot commented on KAFKA-4345:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2064


> Run decktape test for each pull request
> ---
>
> Key: KAFKA-4345
> URL: https://issues.apache.org/jira/browse/KAFKA-4345
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.10.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>
> As of now the ducktape tests that we have for kafka and not run for pull 
> request. We can run these test using travis-ci.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4345) Run decktape test for each pull request

[Sriharsha Chintalapani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani resolved KAFKA-4345.
---
Resolution: Fixed

Issue resolved by pull request 2064
[https://github.com/apache/kafka/pull/2064]

> Run decktape test for each pull request
> ---
>
> Key: KAFKA-4345
> URL: https://issues.apache.org/jira/browse/KAFKA-4345
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.10.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Raghav Kumar Gautam
> Fix For: 0.10.2.0
>
>
> As of now the ducktape tests that we have for kafka and not run for pull 
> request. We can run these test using travis-ci.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2064: KAFKA-4345: Run decktape test for each pull reques...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2064


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] 0.10.1.1 Plan

[Bernard Leach]

Hi Guozhang,

I have added KAFKA-4438 to that list as that would enable publishing the scala 
2.12 builds of 0.10.1.1.  There are other tasks in order to actually publish a 
2.12 but merging that change would enable that process.  There’s a 
corresponding PR on github that consists of a cherry-pick of the corresponding 
change from trunk.

cheers,
bern

> On 24 Nov 2016, at 06:58, Guozhang Wang  wrote:
> 
> Hi everyone,
> 
> We have resolved 15 JIRAs including a few critical bugs in the 0.10.1
> branch since 0.10.1.0 was released so I'd like to propose to release
> 0.10.1.1 soon:
> 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%200.10.1.1%20ORDER%20BY%20priority%20DESC%2C%20key%20DESC
> 
> There are still a few outstanding issues that are not unresolved and have
> 0.10.1.1 as the target version:
> 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%200.10.1.1%20ORDER%20BY%20priority%20DESC%2C%20key%20DESC
> 
> Once we have resolved these issues I will start the release process.
> 
> -- 
> -- Guozhang

[jira] [Commented] (KAFKA-4424) Make serializer classes final

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691914#comment-15691914
 ] 

ASF GitHub Bot commented on KAFKA-4424:
---

Github user MatthiasBechtold closed the pull request at:

https://github.com/apache/kafka/pull/2152


> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2152: KAFKA-4424: make serializer classes final

[MatthiasBechtold]

Github user MatthiasBechtold closed the pull request at:

https://github.com/apache/kafka/pull/2152


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4424) Make serializer classes final

[Matthias Bechtold (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Bechtold resolved KAFKA-4424.
--
Resolution: Not A Problem

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4424) Make serializer classes final

[Matthias Bechtold (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691904#comment-15691904
 ] 

Matthias Bechtold commented on KAFKA-4424:
--

I see, thank you for your time.

I did not think right away that the order of tests would matter.

But this was very interesting to say the least, I think now that I was wrong 
about the final keyword there (interesting article from IBM there).

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4439) Add a builder to NetworkClient

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-4439:
--

 Summary: Add a builder to NetworkClient
 Key: KAFKA-4439
 URL: https://issues.apache.org/jira/browse/KAFKA-4439
 Project: Kafka
  Issue Type: Improvement
  Components: network
Reporter: Colin P. McCabe
Priority: Minor


NetworkClient's constructors have too many parameters.  This makes it hard to 
follow what each value is being initialized to.  Instead, let's use the builder 
pattern to clearly identify what each parameter is and provide sensible 
defaults for each.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4331) Kafka Streams resetter is slow because it joins the same group for each topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691882#comment-15691882
 ] 

ASF GitHub Bot commented on KAFKA-4331:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2138


> Kafka Streams resetter is slow because it joins the same group for each topic
> -
>
> Key: KAFKA-4331
> URL: https://issues.apache.org/jira/browse/KAFKA-4331
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0, 0.10.0.1
>Reporter: Roger Hoover
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> The resetter is joining the same group for each topic which takes ~10secs in 
> my testing.  This makes the reset very slow when you have a lot of topics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2138: KAFKA-4331: Kafka Streams resetter is slow because...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2138


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691817#comment-15691817
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/24/16 1:03 AM:


The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded/invoked that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual(jump) tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation/test and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code in the blog, and running it with or without final 
implementations makes no difference.

Also i've taken this test from the above blog, for your final and non final 
cases (i've attached to this jira), if you note I've uploaded two versions, one 
with the final being declared and loaded by the JVM first and vice versa. As 
you note in both the implementation loaded first due to the inlined branch will 
be more performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.



was (Author: michael.andre.pearce):
The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded/invoked that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation/test and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code in the blog, and running it with or without final 
implementations makes no difference.

Also i've taken this test from the above blog, for your final and non final 
cases (i've attached to this jira), if you note I've uploaded two versions, one 
with the final being declared and loaded by the JVM first and vice versa. As 
you note in both the implementation loaded first due to the inlined branch will 
be more performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.


> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691817#comment-15691817
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/24/16 12:59 AM:
-

The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded/invoked that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation/test and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code in the blog, and running it with or without final 
implementations makes no difference.

Also i've taken this test from the above blog, for your final and non final 
cases (i've attached to this jira), if you note I've uploaded two versions, one 
with the final being declared and loaded by the JVM first and vice versa. As 
you note in both the implementation loaded first due to the inlined branch will 
be more performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.



was (Author: michael.andre.pearce):
The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded/invoked that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you note in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.


> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691817#comment-15691817
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/24/16 12:57 AM:
-

The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded/invoked that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you note in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.



was (Author: michael.andre.pearce):
The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you note in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.


> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691817#comment-15691817
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/24/16 12:55 AM:
-

The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you note in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.



was (Author: michael.andre.pearce):
The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you not in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.


> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691817#comment-15691817
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/24/16 12:54 AM:
-

The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and move to using 
virtual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you not in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.



was (Author: michael.andre.pearce):
The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and more to using 
visual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you not in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.


> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691817#comment-15691817
 ] 

Michael Andre Pearce (IG) commented on KAFKA-4424:
--

The link you reference re virtual calls.

This is much more about monomorphic call or polymorphic calls. Making a class 
that implements an interface final, where the method invocation is by interface 
methods, does not change this.

This is more to do with the number of class's loaded that implement the 
interface. 

So in case of single implementation being used and loaded your jvm you have a 
monomorphic case for the interface, the JVM will inline this (final or not).

If you happen to have two implementations being used and loaded the jvm will 
still be able to inline but will create a branch case, the second loaded 
implementation will be slower if invoked due to the branch.

If you have more than two implementations loaded the JVM will on loading these 
do on stack replacement of the previously loaded inlined, and more to using 
visual tables.


You'll see this occur if you turn on -XX:+PrintCompilation

A classical implementation and write up showing this is:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note taking the code, and running it with or without final 
implementations makes no difference.

Also i've taken this classical test, for your final and non final cases 
(attached to this jira), if you note I've loaded two versions, one with the 
final being declared and loaded by the JVM first and vice versa. As you not in 
both the implementation loaded first due to the inlined branch will be more 
performant.

On checking your original test case we noted that the FinalByteArraySerializer 
version runs first (due to alphabetic ordering that test are run in) , as such 
it would be always the first in the inline branch benefitting from this, this 
would explain why it seems always final was negligible faster when running your 
benchmark test case.


> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1603: KAFKA-1429: Yet another deadlock in controller shu...

[pengwei-li]

Github user pengwei-li closed the pull request at:

https://github.com/apache/kafka/pull/1603


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4438) BACKPORT - Add scala 2.12 support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691780#comment-15691780
 ] 

ASF GitHub Bot commented on KAFKA-4438:
---

GitHub user leachbj opened a pull request:

https://github.com/apache/kafka/pull/2164

KAFKA-4438; Cross compile to Scala 2.12.0

(cherry picked from commit f3aad3b54b7cbc5109d8398829a31100fd82b3e0)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/leachbj/kafka 0.10.1-2.12-backport

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2164.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2164


commit a7ab863c84119e10c69ababe5393291bfda3c5b4
Author: Bernard Leach 
Date:   2016-11-17T02:58:13Z

KAFKA-4438; Cross compile to Scala 2.12.0

(cherry picked from commit f3aad3b54b7cbc5109d8398829a31100fd82b3e0)




> BACKPORT - Add scala 2.12 support
> -
>
> Key: KAFKA-4438
> URL: https://issues.apache.org/jira/browse/KAFKA-4438
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.10.1.0, 0.10.0.1
>Reporter: Bernard Leach
>Assignee: Bernard Leach
> Fix For: 0.10.1.1
>
>
> Now that Scala 2.12 has now been officially released releasing 2.12 builds of 
> the kafka artifacts will allow downstream projects such as reactive-kafka via 
> scalatest-embedded-kafka to release 2.12 builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2164: KAFKA-4438; Cross compile to Scala 2.12.0

[leachbj]

GitHub user leachbj opened a pull request:

https://github.com/apache/kafka/pull/2164

KAFKA-4438; Cross compile to Scala 2.12.0

(cherry picked from commit f3aad3b54b7cbc5109d8398829a31100fd82b3e0)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/leachbj/kafka 0.10.1-2.12-backport

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2164.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2164


commit a7ab863c84119e10c69ababe5393291bfda3c5b4
Author: Bernard Leach 
Date:   2016-11-17T02:58:13Z

KAFKA-4438; Cross compile to Scala 2.12.0

(cherry picked from commit f3aad3b54b7cbc5109d8398829a31100fd82b3e0)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4438) BACKPORT - Add scala 2.12 support

[Bernard Leach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bernard Leach updated KAFKA-4438:
-
Fix Version/s: (was: 0.10.2.0)
   0.10.1.1

> BACKPORT - Add scala 2.12 support
> -
>
> Key: KAFKA-4438
> URL: https://issues.apache.org/jira/browse/KAFKA-4438
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.10.1.0, 0.10.0.1
>Reporter: Bernard Leach
>Assignee: Bernard Leach
> Fix For: 0.10.1.1
>
>
> Now that Scala 2.12 has now been officially released releasing 2.12 builds of 
> the kafka artifacts will allow downstream projects such as reactive-kafka via 
> scalatest-embedded-kafka to release 2.12 builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4438) BACKPORT - Add scala 2.12 support

[Bernard Leach (JIRA)]

Bernard Leach created KAFKA-4438:


 Summary: BACKPORT - Add scala 2.12 support
 Key: KAFKA-4438
 URL: https://issues.apache.org/jira/browse/KAFKA-4438
 Project: Kafka
  Issue Type: Improvement
  Components: build
Affects Versions: 0.10.1.0, 0.10.0.1
Reporter: Bernard Leach
Assignee: Bernard Leach
 Fix For: 0.10.2.0


Now that Scala 2.12 has now been officially released releasing 2.12 builds of 
the kafka artifacts will allow downstream projects such as reactive-kafka via 
scalatest-embedded-kafka to release 2.12 builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4430) Broker logging "Topic and partition to exceptions: [topic,6] -> kafka.common.MessageSizeTooLargeException"

[Srinivas Dhruvakumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691773#comment-15691773
 ] 

Srinivas Dhruvakumar commented on KAFKA-4430:
-

I figured out the issue. The payload was gzipped. It was getting compressed 
twice and the behavior was erratic. Thanks 

> Broker logging "Topic and partition to exceptions: [topic,6] -> 
> kafka.common.MessageSizeTooLargeException"
> --
>
> Key: KAFKA-4430
> URL: https://issues.apache.org/jira/browse/KAFKA-4430
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.1
> Environment: Production 
>Reporter: Srinivas Dhruvakumar
>  Labels: newbie
>
> I have a setup as below 
> DC Kafka 
> Mirrormaker 
> Aggregate Kafka
> Here is the following settings. I have set the max.message.bytes to 1M Bytes 
> on DC and AGG kafka side. Mirrormaker producer settings --  batch.size is set 
> to 500 K Bytes and max.request.size is set to 1 M Byte and ack to 0 , 
> compression-> gzip . 
> However on the Aggregate Kafka I get the following exception 
> Closing connection due to error during produce request with correlation id 
> 414156659 from client id producer-1 with ack=0
> Topic and partition to exceptions: [topic1,6] -> 
> kafka.common.MessageSizeTooLargeException
> Is this a bug or why would this happen. I have configured mirrormaker to send 
> messages less than 1 M Bytes . Are the messages getting dropped ? Under what 
> circumstances this error occurs ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Andre Pearce (IG) updated KAFKA-4424:
-
Attachment: FinalTestReversed.java
FinalTest.java

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
> Attachments: FinalTest.java, FinalTestReversed.java
>
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Andre Pearce (IG) updated KAFKA-4424:
-
Comment: was deleted

(was: Just reading the link you reference.

Virtual calls and jump tables is monomorphic vs polymorphic calls.

If only one implementation(this implementation being final or not) of an 
interface is loaded then you will get a monomorphic implementation and it can 
be fully inlined. 

If you load another, your method is inlined but with a branch, once you load 
further hotspot will do on stack replacement and this is when jump tables are 
needed.

You can see this occurring with on stack replacement with 
-XX:+PrintCompilation, as classes are loaded.

A classical test case showing this:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note that making the class's final or not in the test, makes no 
difference to the outcome.

In case of serialisers/deserialisers this is our case as we have an interface 
that is implemented, as such making these implementations final doesn't help, 
as all reference by calling classes are invoking methods on the interface, as 
such its performance is about how many implementations are loaded.)

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691736#comment-15691736
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/24/16 12:12 AM:
-

Just reading the link you reference.

Virtual calls and jump tables is monomorphic vs polymorphic calls.

If only one implementation(this implementation being final or not) of an 
interface is loaded then you will get a monomorphic implementation and it can 
be fully inlined. 

If you load another, your method is inlined but with a branch, once you load 
further hotspot will do on stack replacement and this is when jump tables are 
needed.

You can see this occurring with on stack replacement with 
-XX:+PrintCompilation, as classes are loaded.

A classical test case showing this:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note that making the class's final or not in the test, makes no 
difference to the outcome.

In case of serialisers/deserialisers this is our case as we have an interface 
that is implemented, as such making these implementations final doesn't help, 
as all reference by calling classes are invoking methods on the interface, as 
such its performance is about how many implementations are loaded.


was (Author: michael.andre.pearce):
Just reading the link you reference.

Virtual calls and jump tables is monomorphic vs polymorphic calls.

If only one implementation(this implementation being final or not) of an 
interface is loaded then you will get a monomorphic implementation and it can 
be fully inlined. 

If you load another, your method is inlined but with a branch, once you load 
further hotspot will get on stack replacement and this is when jump tables are 
needed.

You can see this occurring with on stack replacement with 
-XX:+PrintCompilation, as classes are loaded.

A classical test case showing this:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note that making the class's final or not in the test, makes no 
difference to the outcome.

In case of serialisers/deserialisers this is our case as we have an interface 
that is implemented, as such making these implementations final doesn't help, 
as all reference by calling classes are invoking methods on the interface, as 
such its performance is about how many implementations are loaded.

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691736#comment-15691736
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/24/16 12:12 AM:
-

Just reading the link you reference.

Virtual calls and jump tables is monomorphic vs polymorphic calls.

If only one implementation(this implementation being final or not) of an 
interface is loaded then you will get a monomorphic implementation and it can 
be fully inlined. 

If you load another, your method is inlined but with a branch, once you load 
further hotspot will get on stack replacement and this is when jump tables are 
needed.

You can see this occurring with on stack replacement with 
-XX:+PrintCompilation, as classes are loaded.

A classical test case showing this:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note that making the class's final or not in the test, makes no 
difference to the outcome.

In case of serialisers/deserialisers this is our case as we have an interface 
that is implemented, as such making these implementations final doesn't help, 
as all reference by calling classes are invoking methods on the interface, as 
such its performance is about how many implementations are loaded.


was (Author: michael.andre.pearce):
Just reading the link you reference.

Virtual calls and jump tables is monomorphic vs polymorphic calls.

If only one implementation(this implementation being final or not) of an 
interface is loaded then you will get a monomorphic implementation and it can 
be fully inlined. 

If you load another, your method is inlined but with a branch, once you load 
further hotspot will get on stack replacement and this is when jump tables are 
needed.

You can see this occurring with on stack replacement with 
-XX:+PrintCompilation, as classes are loaded.

A classical test case showing this:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note that making the class's final or not in the test, makes no 
difference to the outcome.

In case of serialisers/deserialisers this is our case as we have an interface 
that is implemented, as such making these implementations final doesn't do much.

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691736#comment-15691736
 ] 

Michael Andre Pearce (IG) commented on KAFKA-4424:
--

Just reading the link you reference.

Virtual calls and jump tables is monomorphic vs polymorphic calls.

If only one implementation(this implementation being final or not) of an 
interface is loaded then you will get a monomorphic implementation and it can 
be fully inlined. 

If you load another, your method is inlined but with a branch, once you load 
further hotspot will get on stack replacement and this is when jump tables are 
needed.

You can see this occurring with on stack replacement with 
-XX:+PrintCompilation, as classes are loaded.

A classical test case showing this:
http://mechanical-sympathy.blogspot.co.uk/2012/04/invoke-interface-optimisations.html

You'll note that making the class's final or not in the test, makes no 
difference to the outcome.

In case of serialisers/deserialisers this is our case as we have an interface 
that is implemented, as such making these implementations final doesn't do much.

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

[radai]

Hi Jun,

I've added the sensor you requested (or at least I think I did )

On Fri, Nov 18, 2016 at 12:37 PM, Jun Rao  wrote:

> KafkaRequestHandlerPool

[jira] [Created] (KAFKA-4437) Incremental Batch Processing for Kafka Streams

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-4437:
--

 Summary: Incremental Batch Processing for Kafka Streams
 Key: KAFKA-4437
 URL: https://issues.apache.org/jira/browse/KAFKA-4437
 Project: Kafka
  Issue Type: New Feature
  Components: streams
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax


We want to add an “auto stop” feature that terminate a stream application when 
it has processed all the data that was newly available at the time the 
application started (to at current end-of-log, i.e., current high watermark). 
This allows to chop the (infinite) log into finite chunks where each run for 
the application processes one chunk. This feature allows for incremental 
batch-like processing; think "start-process-stop-restart-process-stop-..."

For details see KIP-95: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/23/16 11:30 PM:
-

Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a perfomance boost as people come to think, as the modern jvm 
compilers like hotspot (there is another commercial one which I'm sure we all 
know of ;) but as its propriety/commercial shall avoid it for this discussion 
), they really do a lot of magic under the hood for us.

Please read:
http://www.oracle.com/technetwork/java/whitepaper-135217.html#impact
http://www.oracle.com/technetwork/java/whitepaper-135217.html#optimizations

Whilst this article (by  Brian Goetz - his credentials kinda speak for them 
selves) from way back when is very dated and now not relevant in regards to 
precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

This is a change to API level that becomes more restrictive, then i think 
really this should need a KIP, and no doubt some further tests and arguments on 
a KIP discussion for the pros and con's. 

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.


was (Author: michael.andre.pearce):
Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on 

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/23/16 11:28 PM:
-

Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a perfomance boost as people come to think, as the modern jvm 
compilers like hotspot (there is another commercial one which I'm sure we all 
know of ;) but as its propriety/commercial shall avoid it for this discussion 
), they really do a lot of magic under the hood for us.

Please read:
http://www.oracle.com/technetwork/java/whitepaper-135217.html#impact

Whilst this article (by  Brian Goetz - his credentials kinda speak for them 
selves) from way back when is very dated and now not relevant in regards to 
precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

This is a change to API level that becomes more restrictive, then i think 
really this should need a KIP, and no doubt some further tests and arguments on 
a KIP discussion for the pros and con's. 

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.


was (Author: michael.andre.pearce):
Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand 

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/23/16 11:23 PM:
-

Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a perfomance boost as people come to think, as the modern jvm 
compilers like hotspot (there is another commercial one which I'm sure we all 
know of ;) but as its propriety/commercial shall avoid it for this discussion 
), they really do a lot of magic under the hood for us.

Whilst this article (by  Brian Goetz - his credentials kinda speak for them 
selves) from way back when is very dated and now not relevant in regards to 
precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

This is a change to API level that becomes more restrictive, then i think 
really this should need a KIP, and no doubt some further tests and arguments on 
a KIP discussion for the pros and con's. 

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.


was (Author: michael.andre.pearce):
Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm 

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/23/16 11:22 PM:
-

Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a perfomance boost as people come to think, as the modern jvm 
compilers like hotspot (there is another commercial one which I'm sure we all 
know of ;) but as its propriety/commercial shall avoid it for this discussion 
), they really do a lot of magic under the hood for us.

Whilst this article (by  Brian Goetz - his credentials kinda speak for them 
selves) from way back when is very dated and now not relevant in regards to 
precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

This is a change to API level, then i think really this should need a KIP, and 
no doubt some further tests and arguments on a KIP discussion for the pros and 
con's. 

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.


was (Author: michael.andre.pearce):
Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above 

Re: [DISCUSS] KIP-93: Improve invalid timestamp handling in Kafka Streams

[Matthias J. Sax]

Thanks for the feedback. I updated the KIP for (1) and (2).

However not for (3): Why should it be required to reset an application?
If user processed "good" data with valid timestamps, behavior does not
change. If user tried to process "bad" data with invalid timestamps, the
application does fail currently anyway, so there is nothing to reset.


-Matthias

On 11/22/16 9:53 AM, Guozhang Wang wrote:
> Regarding the "compatibility" section, I would suggest being a bit more
> specific about why it is a breaking change. For Streams, it could mean
> different things:
> 
> 1. User need code change when switching library dependency on the new
> version, otherwise it won't compile(I think this is the case for this KIP).
> 2. User need code change when switching library dependency on the new
> version, otherwise runtime exception will be thrown.
> 3. Existing application state as well as internal topics need to be swiped
> and the program need to restart from zero.
> 
> 
> Guozhang
> 
> On Fri, Nov 18, 2016 at 12:27 PM, Matthias J. Sax 
> wrote:
> 
>> Hi all,
>>
>> I want to start a discussion about KIP-93:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 93%3A+Improve+invalid+timestamp+handling+in+Kafka+Streams
>>
>> Looking forward to your feedback.
>>
>>
>> -Matthias
>>
>>
> 
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/23/16 11:12 PM:
-

Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a perfomance boost as people come to think, as the modern jvm 
compilers like hotspot (there is another commercial one which I'm sure we all 
know of ;) but as its propriety/commercial shall avoid it for this discussion 
), they really do a lot of magic under the hood for us.

Whilst this article (by  Brian Goetz - his credentials kinda speak for them 
selves) from way back when is very dated and now not relevant in regards to 
precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

As stated, this is a change to API level, then i think really this should need 
a KIP, and no doubt some further tests and arguments on a KIP discussion for 
the pros and con's.

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.


was (Author: michael.andre.pearce):
Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/23/16 11:01 PM:
-

Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a perfomance boost as people come to think, as the modern jvm 
compilers like hotspot (there is another commercial one which I'm sure we all 
know of ;) but as its propriety/commercial shall avoid it for this discussion 
), they really do a lot of magic under the hood for us.

Whilst this article from way back when is very dated and now not relevant in 
regards to precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

As stated, this is a change to API level, then i think really this should need 
a KIP, and no doubt some further tests and arguments on a KIP discussion for 
the pros and con's.

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.


was (Author: michael.andre.pearce):
Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop 

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) edited comment on KAFKA-4424 at 11/23/16 10:57 PM:
-

Hi Matthias,

Thanks for putting together some initial tests.

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a speed boost as people come to think, as the modern jvm compilers 
like hotspot (there is another commercial one which I'm sure we all know of ;) 
but as its propriety/commercial shall avoid it for this discussion ), they 
really do a lot of magic under the hood for us.

Whilst this article from way back when is very dated and now not relevant in 
regards to precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

As stated, this is a change to API level, then i think really this should need 
a KIP, and no doubt some further tests and arguments on a KIP discussion for 
the pros and con's.

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.


was (Author: michael.andre.pearce):
Hi Matthias,

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our 

[jira] [Commented] (KAFKA-4424) Make serializer classes final

[Michael Andre Pearce (IG) (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691571#comment-15691571
 ] 

Michael Andre Pearce (IG) commented on KAFKA-4424:
--

Hi Matthias,

When we took your test and ran it this evening (with the same parameters) we 
get:

# Run complete. Total time: 01:40:11

Benchmark   Mode   CntScore  Error  
Units
KafkaBenchmark.testFinalSerializer thrpt  1000 9378.179 ±   26.059  
ops/s
KafkaBenchmark.testFinalSerializerNoFlush  thrpt  1000  1283796.450 ± 4976.711  
ops/s
KafkaBenchmark.testSerializer  thrpt  1000 9325.273 ±   26.581  
ops/s
KafkaBenchmark.testSerializerNoFlush   thrpt  1000  1289296.549 ± 5127.774  
ops/s

The performance difference we are seeing is very negligible at best. We have 
run this across a few machines (1 macbook, 1 cisco blade server, 1 nutanix vm) 
within our company and get similar results between final and non-final. (we 
actually had one result come in from the linux vm running on our nutanix 
clusters where the non-final was negligible faster, we repeated and it reversed 
to have the other negligible faster, but it does show that this seems to be 
negligible).

We have run on all the machines using the latest Kakfa 0.10.1.0 version, on JDK 
1.8.0_112, VM 25.112-b16 and kafka setup locally (as per we understand you have 
done), the macbook was on El Capitan, and the cisco blade and nutanix vm are 
running RHEL7.

The above stats copied are from running in particular on a laptop (but are 
inline with what we've seen also in our server environment) just easier to copy 
from as our server environments are protected.

MacBook Pro (Retina, 15-inch, Mid 2015)
2.2 GHz Intel Core i7
16 GB 1600 MHz DDR3


This is what we have come to expect, essentially making a class final doesn't 
give as much a speed boost as people come to think, as the modern jvm compilers 
like hotspot (there is another commercial one which I'm sure we all know of ;) 
but as its propriety/commercial shall avoid it for this discussion ), they 
really do a lot of magic under the hood for us.

Whilst this article from way back when is very dated and now not relevant in 
regards to precise jvm internals as a lot has moved on. 
http://www.ibm.com/developerworks/java/library/j-jtp1029/index.html

There is one core take away i think is important, and is more important when 
making a decision to use final or not.

"
final classes and methods can be a significant inconvenience when programming 
-- they limit your options for reusing existing code and extending the 
functionality of existing classes. While sometimes a class is made final for a 
good reason, such as to enforce immutability, the benefits of using final 
should outweigh the inconvenience. Performance enhancement is almost always a 
bad reason to compromise good object-oriented design principles, and when the 
performance enhancement is small or nonexistent, this is a bad trade-off indeed.
"

As stated, this is a change to API level, then i think really this should need 
a KIP, and no doubt some further tests and arguments on a KIP discussion for 
the pros and con's.

Its worth noting also very recently the ProducerRecord and ConsumerRecord, for 
extensibility reasons were made non-final, if you take the current master.

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Data (re)processing with Kafka (new wiki page)

[Matthias J. Sax]

Hi,

we added a new wiki page that is supposed to collect data (re)processing
scenario with Kafka:

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Data+(Re)Processing+Scenarios

We added already a couple of scenarios we think might be common and want
to invite all of you to add more. This helps to get a better overview of
requirements to enable new use cases.

We are looking forward to your feedback!


-Matthias



signature.asc
Description: OpenPGP digital signature

[jira] [Created] (KAFKA-4436) Provide builder pattern for StreamsConfig

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-4436:
--

 Summary: Provide builder pattern for StreamsConfig
 Key: KAFKA-4436
 URL: https://issues.apache.org/jira/browse/KAFKA-4436
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax
Priority: Minor


Currently, {{StreamsConfig}} parameters must be set "manually" as key value 
pairs. This has multiple disadvantages from a user point of view:

 - mandatory arguments could be missing
 - data types might be wrong
 - producer/consumer config parameters could conflict as they might have the 
same name (user needs to know to prefix them to avoid conflict)

Those problems have different impact: either a runtime exception is thrown if 
the problem is detected (e.g. missing parameter or wrong type) or the 
application is just not configured correctly (producer/consumer has wrong 
config).

A builder pattern would avoid those problems by forcing the user in the first 
place to specify thing correctly (otherwise, it won't compile). For example 
something like this:

{noformat}
StreamsConfig config = StreamsConfig.builder()
.setApplicationId(String appId)
.addBootstrapServer(String host, int port)
.addBootstrapServer(String host, int port)
.addZookeeper(String host, int port)
.addZookeeper(String host, int port)
.setStateDirectory(File path)
.setConsumerConfig(
ConsumerConfig.builder()
.setAutoOffsetReset(...)
.build()
)
.build();
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

FOSDEM 2017 HPC, Bigdata and Data Science DevRoom CFP is closing soon

[Roman Shaposhnik]

Hi!

apologies for the extra wide distribution (this exhausts my once
a year ASF mail-to-all-bigdata-projects quota ;-)) but I wanted
to suggest that all of you should consider submitting talks
to FOSDEM 2017 HPC, Bigdata and Data Science DevRoom:
https://hpc-bigdata-fosdem17.github.io/

It was a great success this year and we hope to make it an even
bigger success in 2017.

Besides -- FOSDEM is the biggest gathering of open source
developers on the face of the earth -- don't miss it!

Thanks,
Roman.

P.S. If you have any questions -- please email me directly and
see you all in Brussels!

[DISCUSS] 0.10.1.1 Plan

[Guozhang Wang]

Hi everyone,

We have resolved 15 JIRAs including a few critical bugs in the 0.10.1
branch since 0.10.1.0 was released so I'd like to propose to release
0.10.1.1 soon:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%200.10.1.1%20ORDER%20BY%20priority%20DESC%2C%20key%20DESC

There are still a few outstanding issues that are not unresolved and have
0.10.1.1 as the target version:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%200.10.1.1%20ORDER%20BY%20priority%20DESC%2C%20key%20DESC

Once we have resolved these issues I will start the release process.

-- 
-- Guozhang

[jira] [Commented] (KAFKA-4178) Replication Throttling: Consolidate Rate Classes

[Ben Stopford (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691178#comment-15691178
 ] 

Ben Stopford commented on KAFKA-4178:
-

Yes, that should be fine. 

> Replication Throttling: Consolidate Rate Classes
> 
>
> Key: KAFKA-4178
> URL: https://issues.apache.org/jira/browse/KAFKA-4178
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
> Fix For: 0.10.2.0
>
>
> Replication throttling is using a different implementation of Rate to client 
> throttling (Rate & SimpleRate). These should be consolidated so both use the 
> same approach. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4435) Improve storage overhead of group metadata

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691169#comment-15691169
 ] 

Jason Gustafson commented on KAFKA-4435:


cc [~onurkaraman]

> Improve storage overhead of group metadata
> --
>
> Key: KAFKA-4435
> URL: https://issues.apache.org/jira/browse/KAFKA-4435
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Jason Gustafson
>
> The GroupMetadataManager serializes the full subscriptions and assignments of 
> all consumer group members for each generation as a single message. This is a 
> problem for large consumer groups with a large number of topics since each 
> member's subscription/assignment is serialized separately. So if you have n 
> consumers each subscribing to the same m topics, then the serialized message 
> will contain m*n subscribed topics. At a certain size, you end up exceeding 
> the max message size.
> Some ideas for getting around this have been 1) turning on compression and 2) 
> adding regex support to the protocol. Both of these help, but maybe we should 
> question whether the subscriptions/assignments need to be written at all. The 
> reason to include this information in the log is basically it prevent a 
> rebalance on coordinator failover. After failover, the new coordinator can 
> consume the log and determine the full state of every group. The consumers in 
> the group simply send heartbeats to the new coordinator, once it is found.
> In fact, preventing the rebalance is not really the main issue: it's ensuring 
> that the last generation can commit its offsets. If nothing were written to 
> the log, then the group would be recreated after failover from scratch and 
> existing members would not be able to commit offsets (since their generation 
> would no longer be valid). But the subscription/assignment is opaque to the 
> coordinator and is not actually used when committing offsets. All it really 
> need is the generation and the list of memberIds. 
> Supposing then that we removed the subscriptions/assignments from the group, 
> but retained the generation/memberId information, one loose end is servicing 
> the DescribeGroup request. After failover, we would no longer have the 
> subscription/assignment information we need to answer that request. One 
> option would be to trigger a rebalance after failover in order to repopulate 
> it. The previous generation would still be able to commit offsets before 
> rejoining the group. Potentially we could even delay this rebalance until we 
> actually receive a DescribeGroups request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4435) Improve storage overhead of group metadata

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4435:
--

 Summary: Improve storage overhead of group metadata
 Key: KAFKA-4435
 URL: https://issues.apache.org/jira/browse/KAFKA-4435
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Jason Gustafson


The GroupMetadataManager serializes the full subscriptions and assignments of 
all consumer group members for each generation as a single message. This is a 
problem for large consumer groups with a large number of topics since each 
member's subscription/assignment is serialized separately. So if you have n 
consumers each subscribing to the same m topics, then the serialized message 
will contain m*n subscribed topics. At a certain size, you end up exceeding the 
max message size.

Some ideas for getting around this have been 1) turning on compression and 2) 
adding regex support to the protocol. Both of these help, but maybe we should 
question whether the subscriptions/assignments need to be written at all. The 
reason to include this information in the log is basically it prevent a 
rebalance on coordinator failover. After failover, the new coordinator can 
consume the log and determine the full state of every group. The consumers in 
the group simply send heartbeats to the new coordinator, once it is found.

In fact, preventing the rebalance is not really the main issue: it's ensuring 
that the last generation can commit its offsets. If nothing were written to the 
log, then the group would be recreated after failover from scratch and existing 
members would not be able to commit offsets (since their generation would no 
longer be valid). But the subscription/assignment is opaque to the coordinator 
and is not actually used when committing offsets. All it really need is the 
generation and the list of memberIds. 

Supposing then that we removed the subscriptions/assignments from the group, 
but retained the generation/memberId information, one loose end is servicing 
the DescribeGroup request. After failover, we would no longer have the 
subscription/assignment information we need to answer that request. One option 
would be to trigger a rebalance after failover in order to repopulate it. The 
previous generation would still be able to commit offsets before rejoining the 
group. Potentially we could even delay this rebalance until we actually receive 
a DescribeGroups request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4039) Exit Strategy: using exceptions instead of inline invocation of exit/halt

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691142#comment-15691142
 ] 

Guozhang Wang commented on KAFKA-4039:
--

[~hachikuji] [~ijuma] If you guys feel it is an important fix to be in 
0.10.1.1, maybe you can review and merge it?

> Exit Strategy: using exceptions instead of inline invocation of exit/halt
> -
>
> Key: KAFKA-4039
> URL: https://issues.apache.org/jira/browse/KAFKA-4039
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Maysam Yabandeh
>Priority: Critical
> Fix For: 0.10.1.1
>
> Attachments: deadlock-stack2
>
>
> The current practice is to directly invoke halt/exit right after the line 
> that intends to terminate the execution. In the case of System.exit this 
> could cause deadlocks if the thread invoking System.exit is holding  a lock 
> that will be requested by the shutdown hook threads that will be started by 
> System.exit. An example is reported by [~aozeritsky] in KAFKA-3924. This 
> would also makes testing more difficult as it would require mocking static 
> methods of System and Runtime classes, which is not natively supported in 
> Java.
> One alternative suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-3924?focusedCommentId=15420269=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15420269]
>  would be to throw some dedicated exceptions that will eventually invoke 
> exit/halt:
> {quote} it would be great to move away from executing `System.exit` inline in 
> favour of throwing an exception (two examples, but maybe we can find better 
> names: FatalExitException and FatalHaltException) that is caught by some 
> central code that then does the `System.exit` or `Runtime.getRuntime.halt`. 
> This helps in a couple of ways:
> (1) Avoids issues with locks being held as in this issue
> (2) It makes it possible to abstract the action, which is very useful in 
> tests. At the moment, we can't easily test for these conditions as they cause 
> the whole test harness to exit. Worse, these conditions are sometimes 
> triggered in the tests and it's unclear why.
> (3) We can have more consistent logging around these actions and possibly 
> extended logging for tests
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1377) transient unit test failure in LogOffsetTest

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-1377:
---
Status: Reopened  (was: Reopened)

> transient unit test failure in LogOffsetTest
> 
>
> Key: KAFKA-1377
> URL: https://issues.apache.org/jira/browse/KAFKA-1377
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Jun Rao
>  Labels: newbie
> Attachments: KAFKA-1377.patch, KAFKA-1377_2014-04-11_17:42:13.patch, 
> KAFKA-1377_2014-04-11_18:14:45.patch
>
>
> Saw the following transient unit test failure.
> kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime FAILED
> junit.framework.AssertionFailedError: expected: but 
> was:
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.failNotEquals(Assert.java:277)
> at junit.framework.Assert.assertEquals(Assert.java:64)
> at junit.framework.Assert.assertEquals(Assert.java:71)
> at 
> kafka.server.LogOffsetTest.testGetOffsetsBeforeEarliestTime(LogOffsetTest.scala:198)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1377) transient unit test failure in LogOffsetTest

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-1377.

Resolution: Fixed

I haven't seen it, so closing it since the original reporter said it was fixed.

> transient unit test failure in LogOffsetTest
> 
>
> Key: KAFKA-1377
> URL: https://issues.apache.org/jira/browse/KAFKA-1377
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Jun Rao
>  Labels: newbie
> Attachments: KAFKA-1377.patch, KAFKA-1377_2014-04-11_17:42:13.patch, 
> KAFKA-1377_2014-04-11_18:14:45.patch
>
>
> Saw the following transient unit test failure.
> kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime FAILED
> junit.framework.AssertionFailedError: expected: but 
> was:
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.failNotEquals(Assert.java:277)
> at junit.framework.Assert.assertEquals(Assert.java:64)
> at junit.framework.Assert.assertEquals(Assert.java:71)
> at 
> kafka.server.LogOffsetTest.testGetOffsetsBeforeEarliestTime(LogOffsetTest.scala:198)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1377) transient unit test failure in LogOffsetTest

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-1377:
---
Fix Version/s: (was: 0.10.1.1)

> transient unit test failure in LogOffsetTest
> 
>
> Key: KAFKA-1377
> URL: https://issues.apache.org/jira/browse/KAFKA-1377
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Jun Rao
>  Labels: newbie
> Attachments: KAFKA-1377.patch, KAFKA-1377_2014-04-11_17:42:13.patch, 
> KAFKA-1377_2014-04-11_18:14:45.patch
>
>
> Saw the following transient unit test failure.
> kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime FAILED
> junit.framework.AssertionFailedError: expected: but 
> was:
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.failNotEquals(Assert.java:277)
> at junit.framework.Assert.assertEquals(Assert.java:64)
> at junit.framework.Assert.assertEquals(Assert.java:71)
> at 
> kafka.server.LogOffsetTest.testGetOffsetsBeforeEarliestTime(LogOffsetTest.scala:198)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4178) Replication Throttling: Consolidate Rate Classes

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4178:
---
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Replication Throttling: Consolidate Rate Classes
> 
>
> Key: KAFKA-4178
> URL: https://issues.apache.org/jira/browse/KAFKA-4178
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
> Fix For: 0.10.2.0
>
>
> Replication throttling is using a different implementation of Rate to client 
> throttling (Rate & SimpleRate). These should be consolidated so both use the 
> same approach. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4178) Replication Throttling: Consolidate Rate Classes

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1569#comment-1569
 ] 

Ismael Juma commented on KAFKA-4178:


[~guozhang], I'd say this can be pushed to 0.10.2.0.

> Replication Throttling: Consolidate Rate Classes
> 
>
> Key: KAFKA-4178
> URL: https://issues.apache.org/jira/browse/KAFKA-4178
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
> Fix For: 0.10.2.0
>
>
> Replication throttling is using a different implementation of Rate to client 
> throttling (Rate & SimpleRate). These should be consolidated so both use the 
> same approach. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4384) ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691103#comment-15691103
 ] 

Ismael Juma commented on KAFKA-4384:


I think so, the PR is close.

> ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted 
> message
> 
>
> Key: KAFKA-4384
> URL: https://issues.apache.org/jira/browse/KAFKA-4384
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
> Environment: Ubuntu 12.04, AWS D2 instance
>Reporter: Jun He
> Fix For: 0.10.1.1
>
> Attachments: KAFKA-4384.patch
>
>
> We recently discovered an issue in Kafka 0.9.0.1 (), where 
> ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted 
> message. As the same logic exists also in Kafka 0.10.0.0 and 0.10.0.1, they 
> may have the similar issue.
> Here are system logs related to this issue.
> 2016-10-30 - 02:33:05,606 ERROR ReplicaFetcherThread-5-1590174474 
> ReplicaFetcherThread.apply - Found invalid messages during fetch for 
> partition [logs,41] offset 39021512238 error Message is corrupt (stored crc = 
> 2028421553, computed crc = 3577227678)
> 2016-10-30 - 02:33:06,582 ERROR ReplicaFetcherThread-5-1590174474 
> ReplicaFetcherThread.error - [ReplicaFetcherThread-5-1590174474], Error due 
> to kafka.common.KafkaException: - error processing data for partition 
> [logs,41] offset 39021512301 Caused - by: java.lang.RuntimeException: Offset 
> mismatch: fetched offset = 39021512301, log end offset = 39021512238.
> First, ReplicaFetcherThread got a corrupted message (offset 39021512238) due 
> to some blip.
> Line 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L138
>  threw exception
> Then, Line 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L145
>  caught it and logged this error.
> Because 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L134
>  updated the topic partition offset to the fetched latest one in 
> partitionMap. So ReplicaFetcherThread skipped the batch with corrupted 
> messages. 
> Based on 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L84,
>  the ReplicaFetcherThread then directly fetched the next batch of messages 
> (with offset 39021512301)
> Next, ReplicaFetcherThread stopped because the log end offset (still 
> 39021512238) didn't match the fetched message (offset 39021512301).
> A quick fix is to move line 134 to be after line 138.
> Would be great to have your comments and please let me know if a Jira issue 
> is needed. Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1377) transient unit test failure in LogOffsetTest

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691106#comment-15691106
 ] 

Guozhang Wang commented on KAFKA-1377:
--

[~ijuma] Is this still an issue? I cannot remember seeing this failure in 
recent Jenkins builds.

> transient unit test failure in LogOffsetTest
> 
>
> Key: KAFKA-1377
> URL: https://issues.apache.org/jira/browse/KAFKA-1377
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Jun Rao
>  Labels: newbie
> Fix For: 0.10.1.1
>
> Attachments: KAFKA-1377.patch, KAFKA-1377_2014-04-11_17:42:13.patch, 
> KAFKA-1377_2014-04-11_18:14:45.patch
>
>
> Saw the following transient unit test failure.
> kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime FAILED
> junit.framework.AssertionFailedError: expected: but 
> was:
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.failNotEquals(Assert.java:277)
> at junit.framework.Assert.assertEquals(Assert.java:64)
> at junit.framework.Assert.assertEquals(Assert.java:71)
> at 
> kafka.server.LogOffsetTest.testGetOffsetsBeforeEarliestTime(LogOffsetTest.scala:198)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2363) ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread Transient Failure

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2363:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread Transient Failure
> ---
>
> Key: KAFKA-2363
> URL: https://issues.apache.org/jira/browse/KAFKA-2363
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Fangmin Lv
>Assignee: Ben Stopford
>  Labels: newbie, transient-unit-test-failure
> Fix For: 0.10.2.0
>
>
> {code}
> kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
> STANDARD_OUT
> [2015-07-24 23:13:05,148] WARN fsync-ing the write ahead log in SyncThread:0 
> took 1084ms which will adversely effect operation latency. See the ZooKeeper 
> troubleshooting guide (org.apache.zookeeper.s
> erver.persistence.FileTxnLog:334)
> kafka.api.ProducerSendTest > testCloseWithZeroTimeoutFromCallerThread FAILED
> java.lang.AssertionError: No request is complete.
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:44)
> at 
> kafka.api.ProducerSendTest$$anonfun$testCloseWithZeroTimeoutFromCallerThread$1.apply$mcVI$sp(ProducerSendTest.scala:343)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at 
> kafka.api.ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread(ProducerSendTest.scala:340)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1573) Transient test failures on LogTest.testCorruptLog

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-1573.
--
Resolution: Not A Problem

> Transient test failures on LogTest.testCorruptLog
> -
>
> Key: KAFKA-1573
> URL: https://issues.apache.org/jira/browse/KAFKA-1573
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Priyank Shah
>  Labels: transient-unit-test-failure
> Fix For: 0.10.1.1
>
>
> Here is an example of the test failure trace:
> junit.framework.AssertionFailedError: expected:<87> but was:<68>
>   at junit.framework.Assert.fail(Assert.java:47)
>   at junit.framework.Assert.failNotEquals(Assert.java:277)
>   at junit.framework.Assert.assertEquals(Assert.java:64)
>   at junit.framework.Assert.assertEquals(Assert.java:130)
>   at junit.framework.Assert.assertEquals(Assert.java:136)
>   at 
> kafka.log.LogTest$$anonfun$testCorruptLog$1.apply$mcVI$sp(LogTest.scala:615)
>   at 
> scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:282)
>   at 
> scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265)
>   at kafka.log.LogTest.testCorruptLog(LogTest.scala:595)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
>   at 
> org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
>   at 
> org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
>   at 
> org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
>   at 
> org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
>   at 
> org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
>   at 
> org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
>   at 
> org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
>   at 
> org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
>   at 
> org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:80)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:47)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:49)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at $Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:103)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
>   at 
> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66)
>   at 
> 

[jira] [Updated] (KAFKA-3689) Exception when attempting to decrease connection count for address with no connections

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3689:
-
Fix Version/s: (was: 0.10.1.1)
   (was: 0.10.0.2)
   0.10.2.0

> Exception when attempting to decrease connection count for address with no 
> connections
> --
>
> Key: KAFKA-3689
> URL: https://issues.apache.org/jira/browse/KAFKA-3689
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
> Environment: ubuntu 14.04,
> java version "1.7.0_95"
> OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.2)
> OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)
> 3 broker cluster (all 3 servers identical -  Intel Xeon E5-2670 @2.6GHz, 
> 8cores, 16 threads 64 GB RAM & 1 TB Disk)
> Kafka Cluster is managed by 3 server ZK cluster (these servers are different 
> from Kafka broker servers). All 6 servers are connected via 10G switch. 
> Producers run from external servers.
>Reporter: Buvaneswari Ramanan
> Fix For: 0.10.2.0
>
> Attachments: KAFKA-3689.log.redacted, kafka-3689-instrumentation.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> As per Ismael Juma's suggestion in email thread to us...@kafka.apache.org 
> with the same subject, I am creating this bug report.
> The following error occurs in one of the brokers in our 3 broker cluster, 
> which serves about 8000 topics. These topics are single partitioned with a 
> replication factor = 3. Each topic gets data at a low rate  – 200 bytes per 
> sec.  Leaders are balanced across the topics.
> Producers run from external servers (4 Ubuntu servers with same config as the 
> brokers), each producing to 2000 topics utilizing kafka-python library.
> This error message occurs repeatedly in one of the servers. Between the hours 
> of 10:30am and 1:30pm on 5/9/16, there were about 10 Million such 
> occurrences. This was right after a cluster restart.
> This is not the first time we got this error in this broker. In those 
> instances, error occurred hours / days after cluster restart.
> =
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. 
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count 
> for address with no connections, address: /X.Y.Z.144 (actual network address 
> masked)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. 
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count 
> for address with no connections, address: /X.Y.Z.144
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at 
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at 
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3875) Transient test failure: kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3875:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Transient test failure: 
> kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime
> 
>
> Key: KAFKA-3875
> URL: https://issues.apache.org/jira/browse/KAFKA-3875
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Ismael Juma
>Assignee: Jun Rao
>  Labels: transient-unit-test-failure
> Fix For: 0.10.2.0
>
>
> It failed in a couple of builds.
> {code}
> java.lang.AssertionError: Should have offset 100 but only successfully sent 0 
> expected:<100> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> kafka.api.BaseProducerSendTest.sendAndVerifyTimestamp(BaseProducerSendTest.scala:228)
>   at 
> kafka.api.BaseProducerSendTest.testSendNonCompressedMessageWithCreateTime(BaseProducerSendTest.scala:170)
> {code}
> And standard out:
> {code}
> org.scalatest.junit.JUnitTestFailedError: Send callback returns the following 
> exception
>   at 
> org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:103)
>   at 
> org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:79)
>   at org.scalatest.Assertions$class.fail(Assertions.scala:1348)
>   at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:79)
>   at 
> kafka.api.BaseProducerSendTest$callback$4$.onCompletion(BaseProducerSendTest.scala:209)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:109)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.maybeExpire(RecordBatch.java:155)
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:245)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:211)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Batch containing 
> 1 record(s) expired due to timeout while requesting metadata from brokers for 
> topic-0
> java.lang.IllegalStateException: Cannot send after the producer is closed.
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:172)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:466)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:430)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:353)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1$$anonfun$onCompletion$1.apply(BaseProducerSendTest.scala:415)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1$$anonfun$onCompletion$1.apply(BaseProducerSendTest.scala:415)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1.onCompletion(BaseProducerSendTest.scala:415)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:107)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:318)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:278)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:57)
>   at 
> org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:364)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> https://jenkins.confluent.io/job/kafka-trunk/905/
> https://jenkins.confluent.io/job/kafka-trunk/919 (the output is similar to 
> the first build)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3986) completedReceives can contain closed channels

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3986:
-
Fix Version/s: (was: 0.10.1.1)
   (was: 0.10.0.2)
   0.10.2.0

> completedReceives can contain closed channels 
> --
>
> Key: KAFKA-3986
> URL: https://issues.apache.org/jira/browse/KAFKA-3986
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Reporter: Ryan P
> Fix For: 0.10.2.0
>
>
> I'm not entirely sure why at this point but it is possible to throw a Null 
> Pointer Exception when processingCompletedReceives. This happens when a 
> fairly substantial number of simultaneously initiated connections are 
> initiated with the server. 
> The processor thread does carry on but it may be worth investigating how the 
> channel could be both closed and  completedReceives. 
> The NPE in question is thrown here:
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala#L490
> It can not be consistently reproduced either. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3832) Kafka Connect's JSON Converter never outputs a null value

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3832:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Kafka Connect's JSON Converter never outputs a null value
> -
>
> Key: KAFKA-3832
> URL: https://issues.apache.org/jira/browse/KAFKA-3832
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Randall Hauch
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.10.2.0
>
>
> Kafka Connect's JSON Converter will never output a null value when 
> {{enableSchemas=true}}. This means that when a connector outputs a 
> {{SourceRecord}} with a null value, the JSON Converter will always produce a 
> message value with:
> {code:json}
> {
>   "schema": null,
>   "payload": null
> }
> {code}
> And, this means that while Kafka log compaction will always be able to remove 
> earlier messages with the same key, log compaction will never remove _all_ of 
> the messages with the same key. 
> The JSON Connector's {{fromConnectData(...)}} should always return null when 
> it is supplied a null value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4178) Replication Throttling: Consolidate Rate Classes

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691096#comment-15691096
 ] 

Guozhang Wang commented on KAFKA-4178:
--

[~benstopford] [~ijuma] Does it need to be in 0.10.1.1?

> Replication Throttling: Consolidate Rate Classes
> 
>
> Key: KAFKA-4178
> URL: https://issues.apache.org/jira/browse/KAFKA-4178
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
> Fix For: 0.10.1.1
>
>
> Replication throttling is using a different implementation of Rate to client 
> throttling (Rate & SimpleRate). These should be consolidated so both use the 
> same approach. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4186) Transient failure in KStreamAggregationIntegrationTest.shouldGroupByKey

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-4186.
--
Resolution: Not A Problem

> Transient failure in KStreamAggregationIntegrationTest.shouldGroupByKey
> ---
>
> Key: KAFKA-4186
> URL: https://issues.apache.org/jira/browse/KAFKA-4186
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Jason Gustafson
>Assignee: Damian Guy
> Fix For: 0.10.1.1
>
>
> Saw this running locally off of trunk:
> {code}
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
> shouldGroupByKey[1] FAILED
> java.lang.AssertionError: Condition not met within timeout 6. Did not 
> receive 10 number of records
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:268)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:211)
> at 
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest.receiveMessages(KStreamAggregationIntegrationTest.java:480)
> at 
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest.shouldGroupByKey(KStreamAggregationIntegrationTest.java:407)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4340:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Change the default value of log.message.timestamp.difference.max.ms to the 
> same as log.retention.ms
> ---
>
> Key: KAFKA-4340
> URL: https://issues.apache.org/jira/browse/KAFKA-4340
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.2.0
>
>
> [~junrao] brought up the following scenario: 
> If users are pumping data with timestamp already passed log.retention.ms into 
> Kafka, the messages will be appended to the log but will be immediately 
> rolled out by log retention thread when it kicks in and the messages will be 
> deleted. 
> To avoid this produce-and-deleted scenario, we can set the default value of 
> log.message.timestamp.difference.max.ms to be the same as log.retention.ms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4182) Move the change logger our of RocksDB stores

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4182:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Move the change logger our of RocksDB stores
> 
>
> Key: KAFKA-4182
> URL: https://issues.apache.org/jira/browse/KAFKA-4182
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>  Labels: performance
> Fix For: 0.10.2.0
>
>
> We currently have the change logger embedded within the RocksDB store 
> implementations, however this results in multiple implementations of the same 
> thing and bad separation of concerns. We should create new LoggedStore that 
> wraps the outer most store when logging is enabled, for example:
> loggedStore -> cachingStore -> meteredStore -> innerStore



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4184) Test failure in ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4184:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Test failure in 
> ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle
> ---
>
> Key: KAFKA-4184
> URL: https://issues.apache.org/jira/browse/KAFKA-4184
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Jason Gustafson
>Assignee: Ben Stopford
> Fix For: 0.10.2.0
>
>
> Seen here: https://builds.apache.org/job/kafka-trunk-jdk7/1544/.
> {code}
> unit.kafka.server.ReplicationQuotasTest > 
> shouldBootstrapTwoBrokersWithFollowerThrottle FAILED
> java.lang.AssertionError: expected:<3.0E7> but was:<2.060725619683527E7>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at 
> unit.kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$14.apply(ReplicationQuotasTest.scala:172)
> at 
> unit.kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$14.apply(ReplicationQuotasTest.scala:168)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at 
> unit.kafka.server.ReplicationQuotasTest.shouldMatchQuotaReplicatingThroughAnAsymmetricTopology(ReplicationQuotasTest.scala:168)
> at 
> unit.kafka.server.ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle(ReplicationQuotasTest.scala:74)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4401) Change the KafkaServerTestHarness and IntegrationTestHarness from trait to abstract class.

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4401:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> Change the KafkaServerTestHarness and IntegrationTestHarness from trait to 
> abstract class.
> --
>
> Key: KAFKA-4401
> URL: https://issues.apache.org/jira/browse/KAFKA-4401
> Project: Kafka
>  Issue Type: Task
>  Components: unit tests
>Affects Versions: 0.10.1.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.2.0
>
>
> The IntegartionTestHarness and KafkaServerTestHarness are useful not only in 
> Kafka unit test, but also useful for the unit tests in other products that 
> depend on Kafka.
> Currently there are two issues making those two test harness classes hard to 
> use by other Java users.
> 1. The two classes are Scala traits. This makes it difficult for people to 
> write Java unit test code. 
> 2. Some of the interfaces are Scala only. 
> It will be good to expose those two classes for more general usage and make 
> them Java friendly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4384) ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691093#comment-15691093
 ] 

Guozhang Wang commented on KAFKA-4384:
--

[~junrao] [~ijuma] Do you think it should be in 0.10.1.1?

> ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted 
> message
> 
>
> Key: KAFKA-4384
> URL: https://issues.apache.org/jira/browse/KAFKA-4384
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
> Environment: Ubuntu 12.04, AWS D2 instance
>Reporter: Jun He
> Fix For: 0.10.1.1
>
> Attachments: KAFKA-4384.patch
>
>
> We recently discovered an issue in Kafka 0.9.0.1 (), where 
> ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted 
> message. As the same logic exists also in Kafka 0.10.0.0 and 0.10.0.1, they 
> may have the similar issue.
> Here are system logs related to this issue.
> 2016-10-30 - 02:33:05,606 ERROR ReplicaFetcherThread-5-1590174474 
> ReplicaFetcherThread.apply - Found invalid messages during fetch for 
> partition [logs,41] offset 39021512238 error Message is corrupt (stored crc = 
> 2028421553, computed crc = 3577227678)
> 2016-10-30 - 02:33:06,582 ERROR ReplicaFetcherThread-5-1590174474 
> ReplicaFetcherThread.error - [ReplicaFetcherThread-5-1590174474], Error due 
> to kafka.common.KafkaException: - error processing data for partition 
> [logs,41] offset 39021512301 Caused - by: java.lang.RuntimeException: Offset 
> mismatch: fetched offset = 39021512301, log end offset = 39021512238.
> First, ReplicaFetcherThread got a corrupted message (offset 39021512238) due 
> to some blip.
> Line 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L138
>  threw exception
> Then, Line 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L145
>  caught it and logged this error.
> Because 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L134
>  updated the topic partition offset to the fetched latest one in 
> partitionMap. So ReplicaFetcherThread skipped the batch with corrupted 
> messages. 
> Based on 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L84,
>  the ReplicaFetcherThread then directly fetched the next batch of messages 
> (with offset 39021512301)
> Next, ReplicaFetcherThread stopped because the log end offset (still 
> 39021512238) didn't match the fetched message (offset 39021512301).
> A quick fix is to move line 134 to be after line 138.
> Would be great to have your comments and please let me know if a Jira issue 
> is needed. Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4399) Deadlock between cleanupGroupMetadata and offset commit

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691085#comment-15691085
 ] 

Guozhang Wang commented on KAFKA-4399:
--

[~hachikuji] [~ijuma] Do you think it should go in 0.10.1.1?

> Deadlock between cleanupGroupMetadata and offset commit
> ---
>
> Key: KAFKA-4399
> URL: https://issues.apache.org/jira/browse/KAFKA-4399
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Alexey Ozeritskiy
>Priority: Blocker
> Fix For: 0.10.1.1
>
> Attachments: deadlock-stack
>
>
> We have upgraded our clusters to 0.10.1.0 and got deadlock issue.
> We thought it smth like https://issues.apache.org/jira/browse/KAFKA-3994, but 
> patch did not help us and our stacks is different. I think it is other issue.
> Stack traces attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3213) [CONNECT] It looks like we are not backing off properly when reconfiguring tasks

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3213:
-
Fix Version/s: (was: 0.10.1.1)
   0.10.2.0

> [CONNECT] It looks like we are not backing off properly when reconfiguring 
> tasks
> 
>
> Key: KAFKA-3213
> URL: https://issues.apache.org/jira/browse/KAFKA-3213
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Gwen Shapira
>Assignee: Liquan Pei
>Priority: Critical
> Fix For: 0.10.2.0
>
>
> Looking at logs of attempt to reconfigure connector while leader is 
> restarting, I see:
> {code}
> [2016-01-29 20:31:01,799] ERROR IO error forwarding REST request:  
> (org.apache.kafka.connect.runtime.rest.RestServer)
> java.net.ConnectException: Connection refused
> [2016-01-29 20:31:01,802] ERROR Request to leader to reconfigure connector 
> tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error 
> trying to forward REST request: Connection refused
> [2016-01-29 20:31:01,802] ERROR Failed to reconfigure connector's tasks, 
> retrying after backoff: 
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error 
> trying to forward REST request: Connection refused
> [2016-01-29 20:31:01,803] DEBUG Sending POST with input 
> [{"tables":"bar","table.poll.interval.ms":"1000","incrementing.column.name":"id","connection.url":"jdbc:mysql://worker1:3306/testdb?user=root","name":"test-mysql-jdbc","tasks.max":"3","task.class":"io.confluent.connect.jdbc.JdbcSourceTask","poll.interval.ms":"1000","topic.prefix":"test-","connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","mode":"incrementing"},{"tables":"foo","table.poll.interval.ms":"1000","incrementing.column.name":"id","connection.url":"jdbc:mysql://worker1:3306/testdb?user=root","name":"test-mysql-jdbc","tasks.max":"3","task.class":"io.confluent.connect.jdbc.JdbcSourceTask","poll.interval.ms":"1000","topic.prefix":"test-","connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","mode":"incrementing"}]
>  to http://worker2:8083/connectors/test-mysql-jdbc/tasks 
> (org.apache.kafka.connect.runtime.rest.RestServer)
> [2016-01-29 20:31:01,803] ERROR IO error forwarding REST request:  
> (org.apache.kafka.connect.runtime.rest.RestServer)
> java.net.ConnectException: Connection refused
> [2016-01-29 20:31:01,804] ERROR Request to leader to reconfigure connector 
> tasks failed (org.apache.kafka.connect.runtime.distributed.DistributedHerder)
> org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: IO Error 
> trying to forward REST request: Connection refused
> {code}
> Note that it looks like we are retrying every 1ms, while I'd expect a retry 
> every 250ms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4393) Improve invalid/negative TS handling

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4393:
---
Status: Patch Available  (was: In Progress)

> Improve invalid/negative TS handling
> 
>
> Key: KAFKA-4393
> URL: https://issues.apache.org/jira/browse/KAFKA-4393
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.10.2.0
>
>
> Currently, Kafka Streams does not handle invalid/negative timestamps returned 
> from the {{TimestampExtractor}} gracefully, but fails with an exception, 
> because negative timestamps cannot get handled in a meaningful way for any 
> time based (ie, window) operators like window aggregates and joins.
> We want to change Streams to a auto-drop behavior for negative timestamps for 
> those records (without any further user notification about dropped record) to 
> enable users to "step over" those records and keep going (instead of an 
> exception). To guard the user from silently dropping messages by default (and 
> kept current fail-fast behavior), we change the default extractor 
> {{ConsumerRecordTimestampExtractor}} to check the extracted meta-data record 
> timestamp and raise an exception if it is negative. Furthermore, we add a 
> "drop-and-log" extractor, as this seems to be a common behavior user might 
> want to have. For any other behavior, users can still provide a custom 
> TS-Extractor implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4430) Broker logging "Topic and partition to exceptions: [topic,6] -> kafka.common.MessageSizeTooLargeException"

[Srinivas Dhruvakumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690723#comment-15690723
 ] 

Srinivas Dhruvakumar commented on KAFKA-4430:
-

I am a tad bit confused why would message size be bigger than 1MB on the Kafka 
AGG if the mirrormaker batch.size is 500 KB and message.request.size is 1MB ? 
Coz Max.request.size checks for the seriailized message size. and i have set 
the batch size taking compression into factor. In worst case if compression 
factor ->1 the size would still be under 1MB.

> Broker logging "Topic and partition to exceptions: [topic,6] -> 
> kafka.common.MessageSizeTooLargeException"
> --
>
> Key: KAFKA-4430
> URL: https://issues.apache.org/jira/browse/KAFKA-4430
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.1
> Environment: Production 
>Reporter: Srinivas Dhruvakumar
>  Labels: newbie
>
> I have a setup as below 
> DC Kafka 
> Mirrormaker 
> Aggregate Kafka
> Here is the following settings. I have set the max.message.bytes to 1M Bytes 
> on DC and AGG kafka side. Mirrormaker producer settings --  batch.size is set 
> to 500 K Bytes and max.request.size is set to 1 M Byte and ack to 0 , 
> compression-> gzip . 
> However on the Aggregate Kafka I get the following exception 
> Closing connection due to error during produce request with correlation id 
> 414156659 from client id producer-1 with ack=0
> Topic and partition to exceptions: [topic1,6] -> 
> kafka.common.MessageSizeTooLargeException
> Is this a bug or why would this happen. I have configured mirrormaker to send 
> messages less than 1 M Bytes . Are the messages getting dropped ? Under what 
> circumstances this error occurs ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4424) Make serializer classes final

[Matthias Bechtold (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690654#comment-15690654
 ] 

Matthias Bechtold edited comment on KAFKA-4424 at 11/23/16 4:50 PM:


Thank you for your feedback, I can totally see your point.

So I did some tests concerning the *producer* throughput, using JMH:

https://github.com/MatthiasBechtold/kafka-throughput-test

I think performance somewhat varies based on test envrionment/method.

Anyways I experienced a small performance gain after 500 seconds warmup 
(between 0.26% - 5.5%) , which is of course somewhat greater in warmup phase 
(this also helps, doesn't it?).

This is not an end-to-end test as proposed, but I did not figure out how to 
quickly implement such test. This test should also provide at least some 
evidence towards possible performance gains.


was (Author: matthiasbechtold):
Thank you for your feedback, I can totally see your point.

So I did some tests concerning the *producer* throughput, using JMH:

https://github.com/MatthiasBechtold/kafka-throughput-test

I think performance somewhat varies based on test envrionment/method.

Anyways I experienced a small performance gain (between 0,26% - 5,5%) , which 
is of course somewhat greater in warmup phase (this also helps, doesn't it?).

This is not an end-to-end test as proposed, but I did not figure out how to 
quickly implement such test. This test should also provide at least some 
evidence towards possible performance gains.

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1429) Yet another deadlock in controller shutdown

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1429.

Resolution: Duplicate
  Assignee: (was: Neha Narkhede)

This is already fixed by KAFKA-4360.

> Yet another deadlock in controller shutdown
> ---
>
> Key: KAFKA-1429
> URL: https://issues.apache.org/jira/browse/KAFKA-1429
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8.1
>Reporter: Dmitry Bugaychenko
> Attachments: kafka_0.9.0.0_controller_dead_lock.patch
>
>
> Found one more case of deadlock in controller during shutdown:
> {code}
> ZkClient-EventThread-57-192.168.41.148:2181,192.168.36.250:2181,192.168.41.207:2181
>  id=57 state=TIMED_WAITING
> - waiting on <0x288a66ec> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> - locked <0x288a66ec> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at 
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468)
> at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:88)
> at 
> kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:339)
> at 
> kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:337)
> at 
> kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:337)
> at kafka.utils.Utils$.inLock(Utils.scala:538)
> at 
> kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:337)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply$mcZ$sp(KafkaController.scala:1068)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1067)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1067)
> at kafka.utils.Utils$.inLock(Utils.scala:538)
> at 
> kafka.controller.KafkaController$SessionExpirationListener.handleNewSession(KafkaController.scala:1067)
> at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> Locked synchronizers: count = 1
>   - java.util.concurrent.locks.ReentrantLock$NonfairSync@22b9b31a
> kafka-scheduler-0 id=172 state=WAITING
> - waiting on <0x22b9b31a> (a 
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> - locked <0x22b9b31a> (a 
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>  owned by 
> ZkClient-EventThread-57-192.168.41.148:2181,192.168.36.250:2181,192.168.41.207:2181
>  id=57
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at 
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at kafka.utils.Utils$.inLock(Utils.scala:536)
> at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1110)
> at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1108)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1108)
> at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4.apply(KafkaController.scala:1087)
> at 

[jira] [Commented] (KAFKA-4424) Make serializer classes final

[Matthias Bechtold (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690654#comment-15690654
 ] 

Matthias Bechtold commented on KAFKA-4424:
--

Thank you for your feedback, I can totally see your point.

So I did some tests concerning the *producer* throughput, using JMH:

https://github.com/MatthiasBechtold/kafka-throughput-test

I think performance somewhat varies based on test envrionment/method.

Anyways I experienced a small performance gain (between 0,26% - 5,5%) , which 
is of course somewhat greater in warmup phase (this also helps, doesn't it?).

This is not an end-to-end test as proposed, but I did not figure out how to 
quickly implement such test. This test should also provide at least some 
evidence towards possible performance gains.

> Make serializer classes final
> -
>
> Key: KAFKA-4424
> URL: https://issues.apache.org/jira/browse/KAFKA-4424
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Matthias Bechtold
>Priority: Minor
>
> Implementations of simple serializers / deserializers should be final to 
> prevent JVM method call overhead.
> See also:
> https://wiki.openjdk.java.net/display/HotSpot/VirtualCalls
> This breaks the API slightly, inheritors must change to generic interfaces 
> Serializer / Deserializer. But architecture-wise final serialization classes 
> make the most sense to me.
> So what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[derek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

derek updated KAFKA-2729:
-
Comment: was deleted

(was: I'm on 0.10.1.0 and seeing the same thing. Maybe related to [~cmolter] is 
saying above, what we see in the logs just prior to a broker becoming 
under-replicated is a flurry of

{noformat}
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is 
not the leader for that topic-partition.
{noformat}

messages. After that we see a bunch of activity around adding and removing 
fetchers, then it goes into the infinite ISR shrink loop. The only way we can 
recover is to restart.)

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[derek (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690420#comment-15690420
 ] 

derek commented on KAFKA-2729:
--

I'm on 0.10.1.0 and seeing the same thing. Maybe related to [~cmolter] is 
saying above, what we see in the logs just prior to a broker becoming 
under-replicated is a flurry of

{noformat}
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is 
not the leader for that topic-partition.
{noformat}

messages. After that we see a bunch of activity around adding and removing 
fetchers, then it goes into the infinite ISR shrink loop. The only way we can 
recover is to restart.

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[derek (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690421#comment-15690421
 ] 

derek commented on KAFKA-2729:
--

I'm on 0.10.1.0 and seeing the same thing. Maybe related to [~cmolter] is 
saying above, what we see in the logs just prior to a broker becoming 
under-replicated is a flurry of

{noformat}
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is 
not the leader for that topic-partition.
{noformat}

messages. After that we see a bunch of activity around adding and removing 
fetchers, then it goes into the infinite ISR shrink loop. The only way we can 
recover is to restart.

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> ---
>
> Key: KAFKA-2729
> URL: https://issues.apache.org/jira/browse/KAFKA-2729
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, 
> we started seeing a large number of undereplicated partitions. The zookeeper 
> cluster recovered, however we continued to see a large number of 
> undereplicated partitions. Two brokers in the kafka cluster were showing this 
> in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for 
> partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 
> (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition 
> [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] 
> not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered 
> after a restart. Our own investigation yielded nothing, I was hoping you 
> could shed some light on this issue. Possibly if it's related to: 
> https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 
> 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

A strange controller log in Kafka 0.9.0.1

[Json Tu]

Hi,
We have a cluster of kafka 0.9.0.1 with 3 nodes, and we found a strange 
controller log as below.

[2016-11-07 03:14:48,575] INFO [SessionExpirationListener on 100], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2016-11-07 03:14:48,578] DEBUG [Controller 100]: Controller resigning, broker 
id 100 (kafka.controller.KafkaController)
[2016-11-07 03:14:48,579] DEBUG [Controller 100]: De-registering 
IsrChangeNotificationListener (kafka.controller.KafkaController)
[2016-11-07 03:14:48,579] INFO [delete-topics-thread-100], Shutting down 
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2016-11-07 03:14:48,579] INFO [delete-topics-thread-100], Stopped  
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2016-11-07 03:14:48,579] INFO [delete-topics-thread-100], Shutdown completed 
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2016-11-07 03:14:48,580] INFO [Partition state machine on Controller 100]: 
Stopped partition state machine (kafka.controller.PartitionStateMachine)
[2016-11-07 03:14:48,580] INFO [Replica state machine on controller 100]: 
Stopped replica state machine (kafka.controller.ReplicaStateMachine)
[2016-11-07 03:14:48,583] INFO [Controller-100-to-broker-101-send-thread], 
Shutting down (kafka.controller.RequestSendThread)
[2016-11-07 03:14:48,584] INFO [Controller-100-to-broker-101-send-thread], 
Stopped  (kafka.controller.RequestSendThread)
[2016-11-07 03:14:48,584] INFO [Controller-100-to-broker-101-send-thread], 
Shutdown completed (kafka.controller.RequestSendThread)
[2016-11-07 03:14:48,586] INFO [Controller-100-to-broker-100-send-thread], 
Shutting down (kafka.controller.RequestSendThread)
[2016-11-07 03:14:48,587] INFO [Controller-100-to-broker-100-send-thread], 
Stopped  (kafka.controller.RequestSendThread)
[2016-11-07 03:14:48,587] INFO [Controller-100-to-broker-100-send-thread], 
Shutdown completed (kafka.controller.RequestSendThread)
[2016-11-07 03:14:48,587] INFO [Controller 100]: Broker 100 resigned as the 
controller (kafka.controller.KafkaController)
[2016-11-07 03:14:48,652] DEBUG [IsrChangeNotificationListener] Fired!!! 
(kafka.controller.IsrChangeNotificationListener)
[2016-11-07 03:14:48,668] INFO [BrokerChangeListener on Controller 100]: Broker 
change listener fired for path /brokers/ids with children 101,100 
(kafka.controller.ReplicaStateMachine$BrokerChangeListener)
[2016-11-07 03:14:48,683] DEBUG [DeleteTopicsListener on 100]: Delete topics 
listener fired for topics  to be deleted 
(kafka.controller.PartitionStateMachine$DeleteTopicsListener)
[2016-11-07 03:14:48,687] INFO [AddPartitionsListener on 100]: Add Partition 
triggered 
{"version":1,"partitions":{"4":[102,101],"5":[100,102],"1":[102,100],"0":[101,102],"2":[100,101],"3":[101,100]}}
 for path /brokers/topics/movie.gateway.merselllog.syncCinema 
(kafka.controller.PartitionStateMachine$AddPartitionsListener)
[2016-11-07 03:14:48,694] INFO [AddPartitionsListener on 100]: Add Partition 
triggered 
{"version":1,"partitions":{"4":[102,101],"5":[100,102],"1":[102,100],"0":[101,102],"2":[100,101],"3":[101,100]}}
 for path /brokers/topics/push_3rdparty_high 
(kafka.controller.PartitionStateMachine$AddPartitionsListener)
[2016-11-07 03:14:48,707] INFO [AddPartitionsListener on 100]: Add Partition 
triggered 
{"version":1,"partitions":{"4":[101,102],"5":[102,100],"1":[101,100],"0":[100,102],"2":[102,101],"3":[100,101]}}
 for path /brokers/topics/icb_msg_push_high_02 
(kafka.controller.PartitionStateMachine$AddPartitionsListener)
[2016-11-07 03:14:48,715] INFO [AddPartitionsListener on 100]: Add Partition 
triggered 
{"version":1,"partitions":{"4":[102,100],"5":[100,101],"1":[102,101],"0":[101,100],"2":[100,102],"3":[101,102]}}
 for path /brokers/topics/movie.gateway.merselllog.unlockSeat 
(kafka.controller.PartitionStateMachine$AddPartitionsListener)


From the log we can see that old controller 100 resigned as the 
controller successfully，but what confused me is that it can also receive 
Fired!!! from IsrChangeNotificationListener which have beed de-register before,
and we can see broker 100 not elect as new controller next time. but we can see 
IsrChangeNotificationListener、DeleteTopicsListener、AddPartitionsListener all 
fired after ressign，does it seems something run with zookeeper.
Any suggestion is appreciated, thanks in advance.

[jira] [Updated] (KAFKA-4379) Remove caching of dirty and removed keys from StoreChangeLogger

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4379:
-
Fix Version/s: 0.10.1.1

> Remove caching of dirty and removed keys from StoreChangeLogger
> ---
>
> Key: KAFKA-4379
> URL: https://issues.apache.org/jira/browse/KAFKA-4379
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Minor
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> The StoreChangeLogger currently keeps a cache of dirty and removed keys and 
> will batch the changelog records such that we don't send a record for each 
> update. However, with KIP-63 this is unnecessary as the batching and 
> de-duping is done by the caching layer. Further, the StoreChangeLogger relies 
> on context.timestamp() which is likely to be incorrect when caching is enabled



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4355) StreamThread intermittently dies with "Topic not found during partition assignment" when broker restarted

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4355:
-
Fix Version/s: 0.10.1.1

> StreamThread intermittently dies with "Topic not found during partition 
> assignment" when broker restarted
> -
>
> Key: KAFKA-4355
> URL: https://issues.apache.org/jira/browse/KAFKA-4355
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0, 0.10.0.0
> Environment: kafka 0.10.0.0
> kafka 0.10.1.0
> uname -a
> Linux lp02485 4.4.0-34-generic #53~14.04.1-Ubuntu SMP Wed Jul 27 16:56:40 UTC 
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> java -version
> java version "1.8.0_92"
> Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
>Reporter: Michal Borowiecki
>Assignee: Eno Thereska
>  Labels: architecture
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> When (a) starting kafka streams app before the broker or
> (b) restarting the broker while the streams app is running:
> the stream thread intermittently dies with "Topic not found during partition 
> assignment" StreamsException.
> This happens about between one in 5 or one in 10 times.
> Stack trace:
> {noformat}
> Exception in thread "StreamThread-2" 
> org.apache.kafka.streams.errors.StreamsException: Topic not found during 
> partition assignment: scheduler
>   at 
> org.apache.kafka.streams.processor.DefaultPartitionGrouper.maxNumPartitions(DefaultPartitionGrouper.java:81)
>   at 
> org.apache.kafka.streams.processor.DefaultPartitionGrouper.partitionGroups(DefaultPartitionGrouper.java:55)
>   at 
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:370)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:313)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:467)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1000(AbstractCoordinator.java:88)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:419)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:395)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:742)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:722)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:186)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:149)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:116)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:479)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:316)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:256)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:308)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:277)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:259)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:407)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
> {noformat}
> Our app has 2 streams in it, consuming from 2 different topics.
> Sometimes the exception happens on both stream threads. Sometimes only on one 
> of the stream threads.
> The exception is preceded by:
> {noformat}
> [2016-10-28 16:17:55,239] INFO [StreamThread-2] (Re-)joining group 
> pool-scheduler 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2016-10-28 16:17:55,240] INFO 

[jira] [Updated] (KAFKA-4311) Multi layer cache eviction causes forwarding to incorrect ProcessorNode

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4311:
-
Fix Version/s: 0.10.1.1

> Multi layer cache eviction causes forwarding to incorrect ProcessorNode 
> 
>
> Key: KAFKA-4311
> URL: https://issues.apache.org/jira/browse/KAFKA-4311
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> The two exceptions below were reported by Frank on the dev mailing list. 
> After investigation, the root cause is multiple cache evictions happening in 
> the same topology. 
> Given a topology like the one below. If a record arriving in `tableOne` 
> causes a cache eviction, it will trigger the `leftJoin` that will do a `get` 
> from `reducer-store`. If the key is not currently cached in `reducer-store`, 
> but is in the backing store, it will be put into the cache, and it may also 
> trigger an eviction. If it does trigger an eviction and the eldest entry is 
> dirty it will flush the dirty keys. It is at this point that the exception in 
> the comment happens (ClassCastException). This occurs because the 
> ProcessorContext is still set to the context of the `leftJoin` and the next 
> child in the topology is `mapValues`.
> We need to set the correct `ProcessorNode`, on the context,  in the 
> `ForwardingCacheFlushListener` prior to calling `context.forward`. We also 
> need to set remember to reset the `ProcessorNode` to the previous node once 
> `context.forward` has completed.
> {code}
> final KTable one = builder.table(Serdes.String(), 
> Serdes.String(), tableOne, tableOne);
> final KTable two = builder.table(Serdes.Long(), 
> Serdes.String(), tableTwo, tableTwo);
> final KTable reduce = two.groupBy(new 
> KeyValueMapper>() {
> @Override
> public KeyValue apply(final Long key, final String 
> value) {
> return new KeyValue<>(value, key);
> }
> }, Serdes.String(), Serdes.Long())
> .reduce(new Reducer() {
> @Override
> public Long apply(final Long value1, final Long value2) {
> return value1 + value2;
> }
> }, new Reducer() {
> @Override
> public Long apply(final Long value1, final Long value2) {
> return value1 - value2;
> }
> }, "reducer-store");
> one.leftJoin(reduce, new ValueJoiner() {
> @Override
> public String apply(final String value1, final Long value2) {
> return value1 + ":" + value2;
> }
> })
> .mapValues(new ValueMapper() {
> @Override
> public String apply(final String value) {
> return value;
> }
> });
> {code}
> This exception is actually a symptom of the exception reported below in the 
> comment. After the first exception is thrown, the StreamThread triggers a 
> shutdown that then throws this exception.
> [StreamThread-1] ERROR
> org.apache.kafka.streams.processor.internals.StreamThread - stream-thread
> [StreamThread-1] Failed to close state manager for StreamTask 0_0:
> org.apache.kafka.streams.errors.ProcessorStateException: task [0_0] Failed
> to close state store addr-organization
> at
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.close(ProcessorStateManager.java:342)
> at
> org.apache.kafka.streams.processor.internals.AbstractTask.closeStateManager(AbstractTask.java:121)
> at
> org.apache.kafka.streams.processor.internals.StreamThread$2.apply(StreamThread.java:341)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.performOnAllTasks(StreamThread.java:322)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.closeAllStateManagers(StreamThread.java:338)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:299)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:262)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:245)
> Caused by: java.lang.IllegalStateException: Key found in dirty key set, but
> entry is null
> at
> org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:112)
> at
> org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:100)
> at
> 

[jira] [Updated] (KAFKA-4361) Streams does not respect user configs for "default" params

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4361:
-
Fix Version/s: 0.10.1.1

> Streams does not respect user configs for "default" params
> --
>
> Key: KAFKA-4361
> URL: https://issues.apache.org/jira/browse/KAFKA-4361
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Roger Hoover
>Assignee: Damian Guy
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> For the config params in CONSUMER_DEFAULT_OVERRIDES 
> (https://github.com/apache/kafka/blob/0.10.1/streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java#L274)
>  such as consumer.max.poll.records and consumer.auto.offset.reset, those 
> parameters are not used and instead overridden by the defaults.  It may not 
> work for some producer config values as well.
> If your application sets those params in the StreamsConfig, they are not used 
> in the underlying Kafka Consumers (and possibly Producers)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4331) Kafka Streams resetter is slow because it joins the same group for each topic

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4331:
-
Fix Version/s: 0.10.1.1

> Kafka Streams resetter is slow because it joins the same group for each topic
> -
>
> Key: KAFKA-4331
> URL: https://issues.apache.org/jira/browse/KAFKA-4331
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0, 0.10.0.1
>Reporter: Roger Hoover
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> The resetter is joining the same group for each topic which takes ~10secs in 
> my testing.  This makes the reset very slow when you have a lot of topics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4275) Check of State-Store-assignment to Processor-Nodes is not enabled

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4275:
-
Fix Version/s: (was: 0.10.1.0)
   0.10.1.1

> Check of State-Store-assignment to Processor-Nodes is not enabled
> -
>
> Key: KAFKA-4275
> URL: https://issues.apache.org/jira/browse/KAFKA-4275
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> In {{ProcessorContextImpl#getStateStores()}} we should check if a store was 
> connected to the processor and thus, if the processor is allowed to access 
> the store. This check is currently disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4269) Multiple KStream instances with at least one Regex source causes NPE when using multiple consumers

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4269:
-
Fix Version/s: 0.10.1.1

> Multiple KStream instances with at least one Regex source causes NPE when 
> using multiple consumers
> --
>
> Key: KAFKA-4269
> URL: https://issues.apache.org/jira/browse/KAFKA-4269
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> I discovered this issue while doing testing for for KAFKA-4114. 
> KAFKA-4131 fixed the issue of a _single_ KStream with a regex source on 
> partitioned topics across multiple consumers.
> //KAFKA-4131 fixed this case assuming an "foo*" topics are partitioned
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();  
> This is a new issue where there are _multiple_
> KStream instances (and one has a regex source) within a single KafkaStreams 
> object. When running the second or "following"
> consumer there are NPE errors generated in the RecordQueue.addRawRecords 
> method when attempting to consume records. 
> For example:
> KStream kstream = builder.source(Pattern.compile("foo.*"));
> KStream kstream2 = builder.source(.): //can be regex or named topic 
> sources
> KafkaStream stream = new KafkaStreams(builder, props);
> stream.start();
> By adding an additional KStream instance like above (whether Regex or Named 
> topic) causes a NPE when run as "follower"
> From my initial debugging I can see the TopicPartition assignments being set 
> on the "follower" KafkaStreams instance, but need to track down why and where 
> all assignments aren't being set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4352) Failure in org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4352:
-
Fix Version/s: 0.10.1.1

> Failure in 
> org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset
> --
>
> Key: KAFKA-4352
> URL: https://issues.apache.org/jira/browse/KAFKA-4352
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.10.2.0
>Reporter: Guozhang Wang
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> We have seen the following scenario happening frequently. It looks similar to 
> KAFKA-2768 which was thought to be fixed.
> {code}
> Stacktrace
> org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
> 'version': java.nio.BufferUnderflowException
>   at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:106)
>   at kafka.admin.AdminClient$$anonfun$1.apply(AdminClient.scala:152)
>   at kafka.admin.AdminClient$$anonfun$1.apply(AdminClient.scala:151)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:151)
>   at 
> org.apache.kafka.streams.integration.ResetIntegrationTest$WaitUntilConsumerGroupGotClosed.conditionMet(ResetIntegrationTest.java:305)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:270)
>   at 
> org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterReset(ResetIntegrationTest.java:135)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> 

[jira] [Updated] (KAFKA-4302) Simplify KTableSource

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4302:
-
Fix Version/s: 0.10.1.1

> Simplify KTableSource
> -
>
> Key: KAFKA-4302
> URL: https://issues.apache.org/jira/browse/KAFKA-4302
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> With the new "interactive queries" feature, source tables are always 
> materialized. Thus, we can remove the stale flag {{KTableSoure#materialized}} 
> (which is always true now) to simply to code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Kafka consumers are not equally distributed

[Ghosh, Achintya (Contractor)]

No, that is not the reason. Initially all the partitions were assigned the 
messages and those were processed very fast and sit idle even other partitions  
are having a lot of messages to be processed.
So I was under impression  that rebalance should be triggered and messages will 
be re-distributed equally again.

Thanks
Achintya 

-Original Message-
From: Sharninder [mailto:sharnin...@gmail.com] 
Sent: Wednesday, November 23, 2016 12:33 AM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka consumers are not equally distributed

Could it be because of the partition key ?

On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < 
achintya_gh...@comcast.com> wrote:

> Hi there,
>
> We are doing the load test in Kafka with 25tps and first 9 hours it 
> went fine almost 80K/hr messages were processed after that we see a 
> lot of lags and we stopped the incoming load.
>
> Currently we see 15K/hr messages are processing. We have 40 consumer 
> instances with concurrency 4 and 2 topics and both is having 160 
> partitions so each consumer with each partition.
>
> What we found that some of the partitions are sitting idle and some of 
> are overloaded and its really slowing down the consumer message processing.
>
> Why rebalancing is not happening and existing messages are not 
> distributed equally among the instances? We tried to restart the app 
> still the same pace. Any idea what could be the reason?
>
> Thanks
> Achintya
>
>


--
--
Sharninder

[jira] [Commented] (KAFKA-4434) KafkaProducer configuration is logged twice

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690203#comment-15690203
 ] 

Ismael Juma commented on KAFKA-4434:


Good catch. Would you like to contribute a fix?

> KafkaProducer configuration is logged twice
> ---
>
> Key: KAFKA-4434
> URL: https://issues.apache.org/jira/browse/KAFKA-4434
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.10.0.1
>Reporter: Ruben de Gooijer
>Priority: Minor
>  Labels: newbie
> Fix For: 0.10.2.0
>
>
> The constructor of org.apache.kafka.clients.producer.KafkaProducer accepts a 
> ProducerConfig which when constructed logs the configuration: 
> https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L58
>  . 
> However, when the construction of KafkaProducer proceeds the provided 
> ProducerConfig is repurposed and another instance is created 
> https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L323
>  which triggers another log with the same contents (only the clientId can 
> differ in case its not supplied in the original config). 
> At first sight this seems like unintended behaviour to me. At least it caused 
> me to dive into it in order to verify if there weren't two producer instances 
> running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4434) KafkaProducer configuration is logged twice

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4434:
---
Fix Version/s: 0.10.2.0

> KafkaProducer configuration is logged twice
> ---
>
> Key: KAFKA-4434
> URL: https://issues.apache.org/jira/browse/KAFKA-4434
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.10.0.1
>Reporter: Ruben de Gooijer
>Priority: Minor
>  Labels: newbie
> Fix For: 0.10.2.0
>
>
> The constructor of org.apache.kafka.clients.producer.KafkaProducer accepts a 
> ProducerConfig which when constructed logs the configuration: 
> https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L58
>  . 
> However, when the construction of KafkaProducer proceeds the provided 
> ProducerConfig is repurposed and another instance is created 
> https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L323
>  which triggers another log with the same contents (only the clientId can 
> differ in case its not supplied in the original config). 
> At first sight this seems like unintended behaviour to me. At least it caused 
> me to dive into it in order to verify if there weren't two producer instances 
> running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4434) KafkaProducer configuration is logged twice

[Ruben de Gooijer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruben de Gooijer updated KAFKA-4434:

Description: 
The constructor of org.apache.kafka.clients.producer.KafkaProducer accepts a 
ProducerConfig which when constructed logs the configuration: 
https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L58
 . 

However, when the construction of KafkaProducer proceeds the provided 
ProducerConfig is repurposed and another instance is created 
https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L323
 which triggers another log with the same contents (only the clientId can 
differ in case its not supplied in the original config). 

At first sight this seems like unintended behaviour to me. At least it caused 
me to dive into it in order to verify if there weren't two producer instances 
running.

  was:
The constructor of org.apache.kafka.clients.producer.KafkaProducer accepts a 
ProducerConfig which when constructed logs the configuration: 
https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L58
 . 

However, when the construction of KafkaProducer proceeds the provided 
ProducerConfig is repurposed and another instance is created 
https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L323
 which triggers another log with the same contents. 

At first sight this seems like unintended behaviour to me. At least it caused 
me to dive into it in order to verify if there weren't two producer instances 
running.


> KafkaProducer configuration is logged twice
> ---
>
> Key: KAFKA-4434
> URL: https://issues.apache.org/jira/browse/KAFKA-4434
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.10.0.1
>Reporter: Ruben de Gooijer
>Priority: Minor
>  Labels: newbie
>
> The constructor of org.apache.kafka.clients.producer.KafkaProducer accepts a 
> ProducerConfig which when constructed logs the configuration: 
> https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L58
>  . 
> However, when the construction of KafkaProducer proceeds the provided 
> ProducerConfig is repurposed and another instance is created 
> https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L323
>  which triggers another log with the same contents (only the clientId can 
> differ in case its not supplied in the original config). 
> At first sight this seems like unintended behaviour to me. At least it caused 
> me to dive into it in order to verify if there weren't two producer instances 
> running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4434) KafkaProducer configuration is logged twice

[Ruben de Gooijer (JIRA)]

Ruben de Gooijer created KAFKA-4434:
---

 Summary: KafkaProducer configuration is logged twice
 Key: KAFKA-4434
 URL: https://issues.apache.org/jira/browse/KAFKA-4434
 Project: Kafka
  Issue Type: Bug
  Components: config
Affects Versions: 0.10.0.1
Reporter: Ruben de Gooijer
Priority: Minor


The constructor of org.apache.kafka.clients.producer.KafkaProducer accepts a 
ProducerConfig which when constructed logs the configuration: 
https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L58
 . 

However, when the construction of KafkaProducer proceeds the provided 
ProducerConfig is repurposed and another instance is created 
https://github.com/apache/kafka/blob/0.10.0.1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L323
 which triggers another log with the same contents. 

At first sight this seems like unintended behaviour to me. At least it caused 
me to dive into it in order to verify if there weren't two producer instances 
running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk8 #1057

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #2163: fix issue: stop kafka or zookeeper failed when ful...

[muliliao]

GitHub user muliliao opened a pull request:

https://github.com/apache/kafka/pull/2163

fix issue: stop kafka or zookeeper failed when full command length reach 
4096 bytes

when user want to stop the zookeeper or kafka server, they will exec the 
script kafka-server-stop.sh or kafka-server-stop.sh.

In kafka-server-stop.sh and kafka-server-stop.sh, it get the pid with ps 
command, as below:
PIDS=$(ps ax | grep -i 'kafka.Kafka' | grep java | grep -v grep | awk 
'{print $1}')
PIDS=$(ps ax | grep java | grep -i QuorumPeerMain | grep -v grep | awk 
'{print $1}')

Since linux limit process command in 4096 bytes, the above command will 
fail when the user install kafka in a  folder with long path, just like 
"/opt/bigdata/kafka_2.10-0.10.0.0".

To fix this issue, we need to modify it as below:
step1. modify kafka-run-class.sh, add -D parameter to the command;
for kafka ,it is : java -Dproc_kafkaServer
for zookeeper, it is : java -Dproc_zookeeper
step2. modify kafka-server-stop.sh and kafka-server-stop.sh, get the PID 
with -D parameter information.
if [ -z "$PIDS" ]; then
PIDS=$(ps ax | grep java | grep -i proc_kafkaServer| grep -v grep | awk 
'{print $1}')
fi
if [ -z "$PIDS" ]; then
PIDS=$(ps ax | grep java | grep -i proc_zookeeper| grep -v grep | awk 
'{print $1}')
fi

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/muliliao/kafka fixed_kafka_stop_error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2163.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2163


commit 4a63862d37afa8885d45ce5379a72093f2fa816e
Author: muliliao 
Date:   2016-11-23T09:12:59Z

full command reach 4096 character issue

if the command reach 4096 character, the stop script can not get the pid, 
and can not stop the service.

commit de93f62463d6e91e056579446517f613ebccad1b
Author: muliliao 
Date:   2016-11-23T09:18:08Z

full command reach 4096 character issue

if the command reach 4096 character, the stop script can not get the pid, 
and can not stop the service.

commit d72565566ea5848c75c788dc9dce2d719b4e7f1e
Author: muliliao 
Date:   2016-11-23T09:20:51Z

full command reach 4096 character issue

if the command reach 4096 character, the stop script can not get the pid, 
and can not stop the service.

commit 3c413bf3069267bac3c2c335430f3f485c6851d5
Author: muliliao 
Date:   2016-11-23T11:24:45Z

full command reach 4096 character issue

if the command reach 4096 character, the stop script can not get the pid, 
and can not stop the service.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2162: Fixed kafka stop error

[muliliao]

Github user muliliao closed the pull request at:

https://github.com/apache/kafka/pull/2162


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4395) KafkaConfig and LogConfig should not have static initialization order dependencies

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4395.

   Resolution: Fixed
Fix Version/s: 0.10.2.0

Issue resolved by pull request 2120
[https://github.com/apache/kafka/pull/2120]

> KafkaConfig and LogConfig should not have static initialization order 
> dependencies
> --
>
> Key: KAFKA-4395
> URL: https://issues.apache.org/jira/browse/KAFKA-4395
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Sumant Tambe
> Fix For: 0.10.2.0, 0.10.1.1
>
>
> LogConfig.configDef.serverDefaultConfigNames is not initialized properly in 
> due to static initialization order dependencies between KafkaConfig and 
> LogConfig. The map ends inserting null values, which are all string literals. 
> Consider the following.
> 1. KafkaConfig begins initialization at first because KafkaServer constructor 
> needs KafkaConfig. 
> 2. at KafkaConfig.LogMessageFormatVersionProp it needs LogConfig. 
> 3. LogConfig begins initialization 
> 4. LogConfig.configDef begins initialization 
> 5. .define(UncleanLeaderElectionEnableProp) needs 
> KafkaConfig.UncleanLeaderElectionEnableProp, which is defined below  
> KafkaConfig.LogMessageFormatVersionProp so it's null 
> 6. Can't start another initialization of KafkaConfig 
> 7. So .define inserts null. This is applicable to all three 
> MinInSyncReplicasProp, UncleanLeaderElectionEnableProp, and 
> CompressionTypeProp



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4395) KafkaConfig and LogConfig should not have static initialization order dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689776#comment-15689776
 ] 

ASF GitHub Bot commented on KAFKA-4395:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2120


> KafkaConfig and LogConfig should not have static initialization order 
> dependencies
> --
>
> Key: KAFKA-4395
> URL: https://issues.apache.org/jira/browse/KAFKA-4395
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Sumant Tambe
> Fix For: 0.10.1.1, 0.10.2.0
>
>
> LogConfig.configDef.serverDefaultConfigNames is not initialized properly in 
> due to static initialization order dependencies between KafkaConfig and 
> LogConfig. The map ends inserting null values, which are all string literals. 
> Consider the following.
> 1. KafkaConfig begins initialization at first because KafkaServer constructor 
> needs KafkaConfig. 
> 2. at KafkaConfig.LogMessageFormatVersionProp it needs LogConfig. 
> 3. LogConfig begins initialization 
> 4. LogConfig.configDef begins initialization 
> 5. .define(UncleanLeaderElectionEnableProp) needs 
> KafkaConfig.UncleanLeaderElectionEnableProp, which is defined below  
> KafkaConfig.LogMessageFormatVersionProp so it's null 
> 6. Can't start another initialization of KafkaConfig 
> 7. So .define inserts null. This is applicable to all three 
> MinInSyncReplicasProp, UncleanLeaderElectionEnableProp, and 
> CompressionTypeProp



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2120: KAFKA-4395: Break static initialization order depe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2120


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---