[jira] [Commented] (KAFKA-2768) New-consumer sends invalid describeGroupResponse while restabilizing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004693#comment-15004693
 ] 

ASF GitHub Bot commented on KAFKA-2768:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/447


> New-consumer sends invalid describeGroupResponse while restabilizing
> 
>
> Key: KAFKA-2768
> URL: https://issues.apache.org/jira/browse/KAFKA-2768
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.9.0.0
>
>
> While a consumer group is restabilizing, it sends describeGroupResponse with 
> no error, but empty metadata and assignments. That leads to SchemaException 
> on client.
> {code}
> Error while executing consumer group command Error reading field 'version': 
> java.nio.BufferUnderflowException
> org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
> 'version': java.nio.BufferUnderflowException
>   at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:106)
>   at 
> kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:163)
>   at 
> kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:161)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:161)
>   at 
> kafka.admin.ConsumerGroupCommand$.describe(ConsumerGroupCommand.scala:109)
>   at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:63)
>   at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2833) OffsetsMessageFormatter hits unexpected exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004800#comment-15004800
 ] 

ASF GitHub Bot commented on KAFKA-2833:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/527


> OffsetsMessageFormatter hits unexpected exception
> -
>
> Key: KAFKA-2833
> URL: https://issues.apache.org/jira/browse/KAFKA-2833
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
>Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
>
> Add the following property to config/consumer.properties 
> exclude.internal.topics=false
> bin/kafka-console-consumer.sh --consumer.config config/consumer.properties 
> --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 
> --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"
> Processed a total of 1 messages
> [2015-11-13 09:17:06,376] ERROR Unknown error when running consumer:  
> (kafka.tools.ConsoleConsumer$)
> org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
> 'metadata': java.nio.BufferUnderflowException
>   at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71)
>   at 
> kafka.coordinator.GroupMetadataManager$.kafka$coordinator$GroupMetadataManager$$readOffsetMessageValue(GroupMetadataManager.scala:861)
>   at 
> kafka.coordinator.GroupMetadataManager$OffsetsMessageFormatter.writeTo(GroupMetadataManager.scala:934)
>   at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:114)
>   at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:65)
>   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:43)
>   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2838) allow comma when specifying superusers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004864#comment-15004864
 ] 

ASF GitHub Bot commented on KAFKA-2838:
---

GitHub user Parth-Brahmbhatt opened a pull request:

https://github.com/apache/kafka/pull/529

KAFKA-2838: Allow comma in super users, allow comma in CLI authz prop…

…erties.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2838

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #529


commit b7b7874dc401e2498b00fae3ab9fb1018d45e572
Author: Parth Brahmbhatt 
Date:   2015-11-13T23:02:10Z

KAFKA-2838: Allow comma in super users, allow comma in CLI authz properties.




> allow comma when specifying superusers
> --
>
> Key: KAFKA-2838
> URL: https://issues.apache.org/jira/browse/KAFKA-2838
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
> Fix For: 0.9.0.0
>
>
> Currently, super users are specified in csv as the following. 
> super.users=User:Bob,User:Alice
> However, if a super user is based on ssl peer principal, it may have comma in 
> the user name like the following.
> CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
> Then, we won't be able to parse the value properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2838) allow comma when specifying superusers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005034#comment-15005034
 ] 

ASF GitHub Bot commented on KAFKA-2838:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/529


> allow comma when specifying superusers
> --
>
> Key: KAFKA-2838
> URL: https://issues.apache.org/jira/browse/KAFKA-2838
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
> Fix For: 0.9.0.0
>
>
> Currently, super users are specified in csv as the following. 
> super.users=User:Bob,User:Alice
> However, if a super user is based on ssl peer principal, it may have comma in 
> the user name like the following.
> CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown
> Then, we won't be able to parse the value properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005772#comment-15005772
 ] 

ASF GitHub Bot commented on KAFKA-2605:
---

GitHub user ZoneMayor opened a pull request:

https://github.com/apache/kafka/pull/531

KAFKA-2605: Replace 'catch: Throwable' clauses with 'NonFatal'

'catch: Throwable' will catch VirtualMachineError, ThreadDeath, 
InterruptedException, LinkageError, ControlThrowable, NotImplementedError; we 
don't want to catch those kind of error

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka 0.8.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #531


commit c466ef240e509c9aed7b2e7c6729df97abe89fff
Author: jinxing 
Date:   2015-11-15T07:11:13Z

KAFKA-2605: Replace 'catch: Throwable' clauses with 'NonFatal'




> Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`
> --
>
> Key: KAFKA-2605
> URL: https://issues.apache.org/jira/browse/KAFKA-2605
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Assignee: jin xing
>  Labels: newbie
>
> The Kafka codebase includes a number of instances where we do `catch t: 
> Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
> NonControl(t)` where `NonFatal` is part of the standard library and 
> `NonControl` is something like:
> {code}
> object NonControl {
>def apply(t: Throwable): Boolean = t match {
>  case _: ControlThrowable => false
>  case _ => true
>}
>   def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else 
> None
> }
> {code}
> We can also use `NonControl` to replace cases like (it's more concise and has 
> the same behaviour):
> {code}
>   case e: ControlThrowable => throw e
>   case e: Throwable => ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005822#comment-15005822
 ] 

ASF GitHub Bot commented on KAFKA-2605:
---

Github user ZoneMayor closed the pull request at:

https://github.com/apache/kafka/pull/531


> Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`
> --
>
> Key: KAFKA-2605
> URL: https://issues.apache.org/jira/browse/KAFKA-2605
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Assignee: jin xing
>  Labels: newbie
>
> The Kafka codebase includes a number of instances where we do `catch t: 
> Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
> NonControl(t)` where `NonFatal` is part of the standard library and 
> `NonControl` is something like:
> {code}
> object NonControl {
>def apply(t: Throwable): Boolean = t match {
>  case _: ControlThrowable => false
>  case _ => true
>}
>   def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else 
> None
> }
> {code}
> We can also use `NonControl` to replace cases like (it's more concise and has 
> the same behaviour):
> {code}
>   case e: ControlThrowable => throw e
>   case e: Throwable => ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2841) Group metadata cache loading is not safe when reloading a partition

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005715#comment-15005715
 ] 

ASF GitHub Bot commented on KAFKA-2841:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/530

KAFKA-2841: safe group metadata cache loading/unloading



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2841

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #530


commit 881380eac954e0906ef2ec0fe3d5d8e067473a35
Author: Jason Gustafson 
Date:   2015-11-14T23:54:25Z

KAFKA-2841: safe group metadata cache loading/unloading




> Group metadata cache loading is not safe when reloading a partition
> ---
>
> Key: KAFKA-2841
> URL: https://issues.apache.org/jira/browse/KAFKA-2841
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> If the coordinator receives a leaderAndIsr request which includes a higher 
> leader epoch for one of the partitions that it owns, then it will reload the 
> offset/metadata for that partition again. This can happen because the leader 
> epoch is incremented for ISR changes which do not result in a new leader for 
> the partition. Currently, the coordinator replaces cached metadata values 
> blindly on reloading, which can result in weird behavior such as unexpected 
> session timeouts or request timeouts while rebalancing.
> To fix this, we need to check that the group being loaded has a higher 
> generation than the cached value before replacing it. Also, if we have to 
> replace a cached value (which shouldn't happen except when loading), we need 
> to be very careful to ensure that any active delayed operations won't affect 
> the group. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005823#comment-15005823
 ] 

ASF GitHub Bot commented on KAFKA-2605:
---

GitHub user ZoneMayor opened a pull request:

https://github.com/apache/kafka/pull/532

KAFKA-2605: Replace 'catch: Throwable' clauses with 'NonFatal'

'catch: Throwable' will catch VirtualMachineError, ThreadDeath, 
InterruptedException, LinkageError, ControlThrowable, NotImplementedError; we 
don't want to catch those kind of error

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka 0.8.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/532.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #532


commit c466ef240e509c9aed7b2e7c6729df97abe89fff
Author: jinxing 
Date:   2015-11-15T07:11:13Z

KAFKA-2605: Replace 'catch: Throwable' clauses with 'NonFatal'




> Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`
> --
>
> Key: KAFKA-2605
> URL: https://issues.apache.org/jira/browse/KAFKA-2605
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Assignee: jin xing
>  Labels: newbie
>
> The Kafka codebase includes a number of instances where we do `catch t: 
> Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
> NonControl(t)` where `NonFatal` is part of the standard library and 
> `NonControl` is something like:
> {code}
> object NonControl {
>def apply(t: Throwable): Boolean = t match {
>  case _: ControlThrowable => false
>  case _ => true
>}
>   def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else 
> None
> }
> {code}
> We can also use `NonControl` to replace cases like (it's more concise and has 
> the same behaviour):
> {code}
>   case e: ControlThrowable => throw e
>   case e: Throwable => ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2605) Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005914#comment-15005914
 ] 

ASF GitHub Bot commented on KAFKA-2605:
---

Github user ZoneMayor closed the pull request at:

https://github.com/apache/kafka/pull/532


> Replace `catch: Throwable` clauses with `NonFatal` or `NonControl`
> --
>
> Key: KAFKA-2605
> URL: https://issues.apache.org/jira/browse/KAFKA-2605
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>Assignee: jin xing
>  Labels: newbie
>
> The Kafka codebase includes a number of instances where we do `catch t: 
> Throwable` where we should really be doing `catch NonFatal(t)` or `catch 
> NonControl(t)` where `NonFatal` is part of the standard library and 
> `NonControl` is something like:
> {code}
> object NonControl {
>def apply(t: Throwable): Boolean = t match {
>  case _: ControlThrowable => false
>  case _ => true
>}
>   def unapply(t: Throwable): Option[Throwable] = if (apply(t)) Some(t) else 
> None
> }
> {code}
> We can also use `NonControl` to replace cases like (it's more concise and has 
> the same behaviour):
> {code}
>   case e: ControlThrowable => throw e
>   case e: Throwable => ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2824) MiniKDC based tests don't run in VirtualBox

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006634#comment-15006634
 ] 

ASF GitHub Bot commented on KAFKA-2824:
---

Github user benstopford closed the pull request at:

https://github.com/apache/kafka/pull/520


> MiniKDC based tests don't run in VirtualBox
> ---
>
> Key: KAFKA-2824
> URL: https://issues.apache.org/jira/browse/KAFKA-2824
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ben Stopford
>
> When running system tests in virtualbox the miniKDC server isn't reachable. 
> Works fine in EC2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2844) Use different keyTab for client and server in SASL tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007118#comment-15007118
 ] 

ASF GitHub Bot commented on KAFKA-2844:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/533

KAFKA-2844; Separate keyTabs for sasl tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka separate-keytabs-for-sasl-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #533


commit 7cd7fba5f3fb23991b93a0180d41b6fd67cd0b99
Author: Ismael Juma 
Date:   2015-11-16T17:10:37Z

Move `FourLetterWords` to its own file and clean-up its usage

commit 4f423a8d6e14660ae25b7c2eba2901e51e75b00e
Author: Ismael Juma 
Date:   2015-11-16T18:51:51Z

Use a different keyTab for server and client in SASL tests

In order to do this, replace templated `kafka_jaas.conf` file
with programmatic approach.




> Use different keyTab for client and server in SASL tests
> 
>
> Key: KAFKA-2844
> URL: https://issues.apache.org/jira/browse/KAFKA-2844
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> We currently use the same keyTab, which could hide problems in the 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2811) Add standby tasks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007355#comment-15007355
 ] 

ASF GitHub Bot commented on KAFKA-2811:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/526


> Add standby tasks
> -
>
> Key: KAFKA-2811
> URL: https://issues.apache.org/jira/browse/KAFKA-2811
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Reporter: Yasuhiro Matsuda
>Assignee: Yasuhiro Matsuda
> Fix For: 0.9.1.0
>
>
> Restoring local state from state change-log topics can be expensive. To 
> alleviate this, we want to have an option to keep replications of local 
> states that are kept up to date. The task assignment logic should be aware of 
> existence of such replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2809) Improve documentation linking

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007469#comment-15007469
 ] 

ASF GitHub Bot commented on KAFKA-2809:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/498


> Improve documentation linking
> -
>
> Key: KAFKA-2809
> URL: https://issues.apache.org/jira/browse/KAFKA-2809
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.2.2
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.0.0
>
>
> Often it is useful to link to a specific header within the documentation. 
> Especially when referencing docs in the mailing lists. 
> This Jira is to add anchors and links for all headers in the docs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2831) kafka-consumer-groups requires zookeeper url when using the new-consumer option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007436#comment-15007436
 ] 

ASF GitHub Bot commented on KAFKA-2831:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/528


> kafka-consumer-groups requires zookeeper url when using the new-consumer 
> option
> ---
>
> Key: KAFKA-2831
> URL: https://issues.apache.org/jira/browse/KAFKA-2831
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jun Rao
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group 
> test-consumer-group --new-consumer --describe
> Exception in thread "main" java.lang.NullPointerException
>   at 
> org.apache.zookeeper.client.ConnectStringParser.(ConnectStringParser.java:50)
>   at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:443)
>   at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
>   at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:69)
>   at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1218)
>   at org.I0Itec.zkclient.ZkClient.(ZkClient.java:155)
>   at org.I0Itec.zkclient.ZkClient.(ZkClient.java:129)
>   at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
>   at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
>   at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:54)
>   at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2845) Add 0.9 clients vs 0.8 brokers compatibility test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007529#comment-15007529
 ] 

ASF GitHub Bot commented on KAFKA-2845:
---

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/537

KAFKA-2845: new client old broker compatibility



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
KAFKA-2845-new-client-old-broker-compatibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #537


commit f8cb7f9ba8f4c6f08d15af4c1df8d439739e15a9
Author: Geoff Anderson 
Date:   2015-10-29T21:43:40Z

Sketch of compatibility test

commit 2c3b65ef7c26fef035b83e44d6ed1f301cc80a34
Author: Geoff Anderson 
Date:   2015-11-16T22:15:46Z

Add test for 0.9 consumer against 0.8 brokers

commit 924a281eeace248f852551d30bf55e8c5d79a7b8
Author: Geoff Anderson 
Date:   2015-11-16T22:40:03Z

Added compatibility test




> Add 0.9 clients vs 0.8 brokers compatibility test
> -
>
> Key: KAFKA-2845
> URL: https://issues.apache.org/jira/browse/KAFKA-2845
> Project: Kafka
>  Issue Type: Task
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Add a simple test or two to document and understand what behavior to expect 
> if users try to run 0.9 java producer or 0.9 scala consumer ("old consumer") 
> against an 0.8.X broker cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2721) Avoid handling duplicate LeaderAndISR requests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007647#comment-15007647
 ] 

ASF GitHub Bot commented on KAFKA-2721:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/436


> Avoid handling duplicate LeaderAndISR requests
> --
>
> Key: KAFKA-2721
> URL: https://issues.apache.org/jira/browse/KAFKA-2721
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Dong Lin
> Fix For: 0.9.0.0
>
>
> Upon controller migration, the new controller will try to resend all the 
> LeaderAndISR requests for any on-going partition reassignments. This can then 
> lead to duplicate such requests sent to the same broker.
> Upon receiving such requests, today brokers do not detect if, for example, it 
> is already the leader for the requested becoming-leader-partition, and does 
> the logic such as 1) stop fetching 2) coordinator migration, etc which is not 
> necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2820) System tests: log level is no longer propagating from service classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007706#comment-15007706
 ] 

ASF GitHub Bot commented on KAFKA-2820:
---

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/538

KAFKA-2820: systest log level

This restores control over log level in KafkaService, and adds SASL debug 
logging when SASL is enabled

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
KAFKA-2820-systest-log-level

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #538


commit 0f27b3b547226d20dc7161acf02753ebbe9b5235
Author: Geoff Anderson 
Date:   2015-11-17T00:38:44Z

Add debug logging for SASL

commit 0fc43371e3094cdb9df336430ed5145490b0010c
Author: Geoff Anderson 
Date:   2015-11-17T00:39:04Z

Restore control over log level in KafkaService




> System tests: log level is no longer propagating from service classes
> -
>
> Key: KAFKA-2820
> URL: https://issues.apache.org/jira/browse/KAFKA-2820
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>
> Many system test service classes specify a log level which should be 
> reflected in the log4j output of the corresponding kafka tools etc.
> However, at least some these log levels are no longer propagating, which 
> makes tests much harder to debug after they have run.
> E.g. KafkaService specifies a DEBUG log level, but all collected log output 
> from brokers is at INFO level or above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2848) Use withClientSslSupport/withClientSaslSupport in Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007911#comment-15007911
 ] 

ASF GitHub Bot commented on KAFKA-2848:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/539

KAFKA-2848: Use client SSL/SASL config utilities in Kafka Connect to avoid 
duplication of configs.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2848-reuse-ssl-sasl-client-configs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #539


commit 752d208cb3b607150cd016cc2776008dda0984b8
Author: Ewen Cheslack-Postava 
Date:   2015-11-17T02:45:53Z

KAFKA-2848: Use client SSL/SASL config utilities in Kafka Connect to avoid 
duplication of configs.




> Use withClientSslSupport/withClientSaslSupport in Kafka Connect
> ---
>
> Key: KAFKA-2848
> URL: https://issues.apache.org/jira/browse/KAFKA-2848
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.1.0
>
>
> These were introduced in KAFKA-2687 and applied to the ProducerConfig and 
> ConsumerConfig classes, but did not get applied to Kafka Connect configs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2848) Use withClientSslSupport/withClientSaslSupport in Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008106#comment-15008106
 ] 

ASF GitHub Bot commented on KAFKA-2848:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/539


> Use withClientSslSupport/withClientSaslSupport in Kafka Connect
> ---
>
> Key: KAFKA-2848
> URL: https://issues.apache.org/jira/browse/KAFKA-2848
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> These were introduced in KAFKA-2687 and applied to the ProducerConfig and 
> ConsumerConfig classes, but did not get applied to Kafka Connect configs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2812) Enhance new consumer integration test coverage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001337#comment-15001337
 ] 

ASF GitHub Bot commented on KAFKA-2812:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/500

KAFKA-2812: improve consumer integration tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2812

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #500


commit ff9a2dfba81e15170204a21dec12307206e9bd61
Author: Jason Gustafson 
Date:   2015-11-10T23:24:04Z

KAFKA-2812: improve consumer integration tests




> Enhance new consumer integration test coverage
> --
>
> Key: KAFKA-2812
> URL: https://issues.apache.org/jira/browse/KAFKA-2812
> Project: Kafka
>  Issue Type: Test
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> There are still some test cases that we didn't get to in KAFKA-2274 
> (including hard broker and client failures) as well as additional validation 
> that can be added to existing test cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2807) Movement of throughput throttler to common broke upgrade tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001357#comment-15001357
 ] 

ASF GitHub Bot commented on KAFKA-2807:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/499


> Movement of throughput throttler to common broke upgrade tests
> --
>
> Key: KAFKA-2807
> URL: https://issues.apache.org/jira/browse/KAFKA-2807
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.9.1.0
>
>
> In order to run compatibility tests with an 0.8.2 producer, and using 
> VerifiableProducer, we use the 0.8.2 kafka-run-tools.sh classpath augmented 
> by the 0.9.0 tools and tools dependencies classpaths.
> Recently, some refactoring efforts moved ThroughputThrottler to 
> org.apache.kafka.common.utils package, but this breaks the existing 
> compatibility tests:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/kafka/common/utils/ThroughputThrottler
> at 
> org.apache.kafka.tools.VerifiableProducer.main(VerifiableProducer.java:334)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.kafka.common.utils.ThroughputThrottler
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 1 more
> {code}
> Given the need to be able to run VerifiableProducer against 0.8.X, I'm not 
> sure VerifiableProducer can depend on org.apache.kafka.common.utils at this 
> point in time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2763) Reduce stream task migrations and initialization costs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001375#comment-15001375
 ] 

ASF GitHub Bot commented on KAFKA-2763:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/497


> Reduce stream task migrations and initialization costs
> --
>
> Key: KAFKA-2763
> URL: https://issues.apache.org/jira/browse/KAFKA-2763
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.0
>Reporter: Yasuhiro Matsuda
>Assignee: Yasuhiro Matsuda
> Fix For: 0.9.1.0
>
>
> Stream task assignment is not aware of either the previous task assignment or 
> local states of participating clients. By making the assignment logic aware 
> of them, we can reduce task migrations and initialization cost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2815) unit test failure in org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002521#comment-15002521
 ] 

ASF GitHub Bot commented on KAFKA-2815:
---

Github user granthenke closed the pull request at:

https://github.com/apache/kafka/pull/510


> unit test failure in 
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest
> -
>
> Key: KAFKA-2815
> URL: https://issues.apache.org/jira/browse/KAFKA-2815
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.1.0
>Reporter: Jun Rao
>
> See the following failure on trunk.
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest
>  > testSubscription FAILED
> java.lang.AssertionError: expected:<[topic1, topic2]> but was:<[topic2, 
> topic1]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest.testSubscription(KafkaStreamingPartitionAssignorTest.java:174)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2815) unit test failure in org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002522#comment-15002522
 ] 

ASF GitHub Bot commented on KAFKA-2815:
---

GitHub user granthenke reopened a pull request:

https://github.com/apache/kafka/pull/510

KAFKA-2815: Fix KafkaStreamingPartitionAssignorTest.testSubscription

Fails when order of elements is incorrect

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka streams-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/510.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #510


commit f6f76ffd622d80e12f753e7175fdf4c08ca2069d
Author: Grant Henke 
Date:   2015-11-12T16:27:44Z

MINOR: Fix KafkaStreamingPartitionAssignorTest.testSubscription

Fails when order of elements is incorrect




> unit test failure in 
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest
> -
>
> Key: KAFKA-2815
> URL: https://issues.apache.org/jira/browse/KAFKA-2815
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.1.0
>Reporter: Jun Rao
>
> See the following failure on trunk.
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest
>  > testSubscription FAILED
> java.lang.AssertionError: expected:<[topic1, topic2]> but was:<[topic2, 
> topic1]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest.testSubscription(KafkaStreamingPartitionAssignorTest.java:174)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2815) unit test failure in org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002533#comment-15002533
 ] 

ASF GitHub Bot commented on KAFKA-2815:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/510


> unit test failure in 
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest
> -
>
> Key: KAFKA-2815
> URL: https://issues.apache.org/jira/browse/KAFKA-2815
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.1.0
>Reporter: Jun Rao
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> See the following failure on trunk.
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest
>  > testSubscription FAILED
> java.lang.AssertionError: expected:<[topic1, topic2]> but was:<[topic2, 
> topic1]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.streams.processor.internals.KafkaStreamingPartitionAssignorTest.testSubscription(KafkaStreamingPartitionAssignorTest.java:174)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2690) Protect passwords from logging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002550#comment-15002550
 ] 

ASF GitHub Bot commented on KAFKA-2690:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/371


> Protect passwords from logging
> --
>
> Key: KAFKA-2690
> URL: https://issues.apache.org/jira/browse/KAFKA-2690
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Jakub Nowak
> Fix For: 0.9.0.0
>
>
> We currently store the key (ssl.key.password), keystore 
> (ssl.keystore.password) and truststore (ssl.truststore.password) passwords as 
> a String in `KafkaConfig`, `ConsumerConfig` and `ProducerConfig`.
> The problem with this approach is that we may accidentally log the password 
> when logging the config.
> A possible solution is to introduce a new `ConfigDef.Type` that overrides 
> `toString` so that the value is hidden.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2809) Improve documentation linking

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000990#comment-15000990
 ] 

ASF GitHub Bot commented on KAFKA-2809:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/498

KAFKA-2809: Improve documentation linking

Often it is useful to link to a specific header within the documentation. 
Especially when referencing docs in the mailing lists.

This adds anchors and links for all headers in the docs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka doc-links

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/498.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #498


commit 3c7f762ec4028906071dfad540311e815bda8355
Author: Grant Henke 
Date:   2015-11-11T19:41:56Z

KAFKA-2809: Improve documentation linking




> Improve documentation linking
> -
>
> Key: KAFKA-2809
> URL: https://issues.apache.org/jira/browse/KAFKA-2809
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.2.2
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.0.0
>
>
> Often it is useful to link to a specific header within the documentation. 
> Especially when referencing docs in the mailing lists. 
> This Jira is to add anchors and links for all headers in the docs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2790) Kafka 0.9.0 doc improvement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000888#comment-15000888
 ] 

ASF GitHub Bot commented on KAFKA-2790:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/491


> Kafka 0.9.0 doc improvement
> ---
>
> Key: KAFKA-2790
> URL: https://issues.apache.org/jira/browse/KAFKA-2790
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Gwen Shapira
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Observed a few issues after uploading the 0.9.0 docs to the Apache site 
> (http://kafka.apache.org/090/documentation.html).
> 1. There are a few places still mentioning 0.8.2.
> docs/api.html:We are in the process of rewritting the JVM clients for Kafka. 
> As of 0.8.2 Kafka includes a newly rewritten Java producer. The next release 
> will include an equivalent Java consumer. These new clients are meant to 
> supplant the existing Scala clients, but for compatability they will co-exist 
> for some time. These clients are available in a seperate jar with minimal 
> dependencies, while the old Scala clients remain packaged with the server.
> docs/api.html:As of the 0.8.2 release we encourage all new development to use 
> the new Java producer. This client is production tested and generally both 
> faster and more fully featured than the previous Scala client. You can use 
> this client by adding a dependency on the client jar using the following 
> example maven co-ordinates (you can change the version numbers with new 
> releases):
> docs/api.html:version0.8.2.0/version
> docs/ops.html:The partition reassignment tool does not have the ability to 
> automatically generate a reassignment plan for decommissioning brokers yet. 
> As such, the admin has to come up with a reassignment plan to move the 
> replica for all partitions hosted on the broker to be decommissioned, to the 
> rest of the brokers. This can be relatively tedious as the reassignment needs 
> to ensure that all the replicas are not moved from the decommissioned broker 
> to only one other broker. To make this process effortless, we plan to add 
> tooling support for decommissioning brokers in 0.8.2.
> docs/quickstart.html: href="https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.2.0/kafka_2.10-0.8.2.0.tgz;
>  title="Kafka downloads">Download the 0.8.2.0 release and un-tar it.
> docs/quickstart.html: tar -xzf kafka_2.10-0.8.2.0.tgz
> docs/quickstart.html: cd kafka_2.10-0.8.2.0
> 2. The generated config tables (broker, producer and consumer) don't have the 
> proper table frames.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2817) Closing an unconnected SslTransport get an invalid close state exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002507#comment-15002507
 ] 

ASF GitHub Bot commented on KAFKA-2817:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/511

KAFKA-2817; Check if socketChannel is connected in `SslTransportLayer.close`

This avoids spurious log warning messages. Also tweak log message
if wrapResult.getStatus != CLOSED.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-2817-unconnected-ssl-transport-layer-close

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/511.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #511


commit 36bd18b0d6b1de88f06227f90fdc020611330964
Author: Ismael Juma 
Date:   2015-11-12T17:31:43Z

Check if socketChannel is connected in `SslTransportLayer.close`

This avoids spurious log warning messages. Also tweak log message
if wrapResult.getStatus != CLOSED.




> Closing an unconnected SslTransport get an invalid close state exception
> 
>
> Key: KAFKA-2817
> URL: https://issues.apache.org/jira/browse/KAFKA-2817
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jun Rao
>Assignee: Ismael Juma
> Fix For: 0.9.0.0
>
>
> Saw the following in our system test. When shutting down a broker, in a short 
> window, the controller can't establish socket connection to the shutdown 
> broker and tries to close it. It hit the following exception. This may leak 
> the underlying socket.
> [2015-11-11 07:24:17,253] WARN Failed to send SSL Close message  
> (org.apache.kafka.common.network.SslTransportLayer)
> java.io.IOException: Invalid close state, will not send network data.
> at 
> org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:156)
> at 
> org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:50)
> at org.apache.kafka.common.network.Selector.close(Selector.java:448)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:316)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:128)
> at 
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
> at 
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:105)
> at 
> kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:58)
> at 
> kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:225)
> at 
> kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:172)
> at 
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:171)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2807) Movement of throughput throttler to common broke upgrade tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001091#comment-15001091
 ] 

ASF GitHub Bot commented on KAFKA-2807:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/499

KAFKA-2807: Move ThroughputThrottler back to tools jar to fix upgrade tests.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2807-relocate-throughput-throttler

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #499


commit 410e876f2466c0fabd2c5e3a1ba9252375a35939
Author: Ewen Cheslack-Postava 
Date:   2015-11-11T20:14:49Z

KAFKA-2807: Move ThroughputThrottler back to tools jar to fix upgrade tests.




> Movement of throughput throttler to common broke upgrade tests
> --
>
> Key: KAFKA-2807
> URL: https://issues.apache.org/jira/browse/KAFKA-2807
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> In order to run compatibility tests with an 0.8.2 producer, and using 
> VerifiableProducer, we use the 0.8.2 kafka-run-tools.sh classpath augmented 
> by the 0.9.0 tools and tools dependencies classpaths.
> Recently, some refactoring efforts moved ThroughputThrottler to 
> org.apache.kafka.common.utils package, but this breaks the existing 
> compatibility tests:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/kafka/common/utils/ThroughputThrottler
> at 
> org.apache.kafka.tools.VerifiableProducer.main(VerifiableProducer.java:334)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.kafka.common.utils.ThroughputThrottler
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 1 more
> {code}
> Given the need to be able to run VerifiableProducer against 0.8.X, I'm not 
> sure VerifiableProducer can depend on org.apache.kafka.common.utils at this 
> point in time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2807) Movement of throughput throttler to common broke upgrade tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002638#comment-15002638
 ] 

ASF GitHub Bot commented on KAFKA-2807:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/512

KAFKA-2807: Fix Kafka Connect packaging and move VerifiableSource/Sink into 
runtime jar.

Gradle does not handle subprojects with the same name (top-level tools vs
connect/tools) properly, making the dependency impossible to express 
correctly
since we need to move the ThroughputThrottler class into the top level tools
project. Moving the current set of tools into the runtime jar works fine 
since
they are only used for system tests at the moment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka kafka-2807-redux

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #512


commit 7632c27cb5583d285aa66a9d2bf0510139666b41
Author: Ewen Cheslack-Postava 
Date:   2015-11-12T18:25:33Z

KAFKA-2807: Fix Kafka Connect packaging and move VerifiableSource/Sink into 
runtime jar.

Gradle does not handle subprojects with the same name (top-level tools vs
connect/tools) properly, making the dependency impossible to express 
correctly
since we need to move the ThroughputThrottler class into the top level tools
project. Moving the current set of tools into the runtime jar works fine 
since
they are only used for system tests at the moment.




> Movement of throughput throttler to common broke upgrade tests
> --
>
> Key: KAFKA-2807
> URL: https://issues.apache.org/jira/browse/KAFKA-2807
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.9.1.0
>
>
> In order to run compatibility tests with an 0.8.2 producer, and using 
> VerifiableProducer, we use the 0.8.2 kafka-run-tools.sh classpath augmented 
> by the 0.9.0 tools and tools dependencies classpaths.
> Recently, some refactoring efforts moved ThroughputThrottler to 
> org.apache.kafka.common.utils package, but this breaks the existing 
> compatibility tests:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/kafka/common/utils/ThroughputThrottler
> at 
> org.apache.kafka.tools.VerifiableProducer.main(VerifiableProducer.java:334)
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.kafka.common.utils.ThroughputThrottler
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 1 more
> {code}
> Given the need to be able to run VerifiableProducer against 0.8.X, I'm not 
> sure VerifiableProducer can depend on org.apache.kafka.common.utils at this 
> point in time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2849) Pass `KafkaChannel.principal` to `RequestChannel.Session`

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008926#comment-15008926
 ] 

ASF GitHub Bot commented on KAFKA-2849:
---

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/541


> Pass `KafkaChannel.principal` to `RequestChannel.Session`
> -
>
> Key: KAFKA-2849
> URL: https://issues.apache.org/jira/browse/KAFKA-2849
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> The current code does:
> {code}
> val session = RequestChannel.Session(new 
> KafkaPrincipal(KafkaPrincipal.USER_TYPE, channel.principal().getName), 
> channel.socketDescription)
> {code}
> It looks to me like we should just pass `channel.principal`.
> cc [~parth.brahmbhatt]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2847) remove principal.builder.class from client configs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008974#comment-15008974
 ] 

ASF GitHub Bot commented on KAFKA-2847:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/542


> remove principal.builder.class from client configs
> --
>
> Key: KAFKA-2847
> URL: https://issues.apache.org/jira/browse/KAFKA-2847
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Ismael Juma
>
> Since only the broker needs to know the principal name, we can remove 
> principal.builder.class from both the producer and the consumer client config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2847) remove principal.builder.class from client configs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008607#comment-15008607
 ] 

ASF GitHub Bot commented on KAFKA-2847:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/542

KAFKA-2847; Remove principal builder class from client configs

Also mark `PrincipalBuilder` as `Unstable` and  tweak docs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-2847-remove-principal-builder-class-from-client-configs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/542.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #542


commit 88ccdfb961ec4adc82121b1ed44a198a4dbe4f89
Author: Ismael Juma 
Date:   2015-11-17T12:48:25Z

Remove principal.builder.class from client configs

commit ce2c307bae8da40bed4eb53cc209c65973b72331
Author: Ismael Juma 
Date:   2015-11-17T12:52:35Z

Mark the `PrincipalBuilder` interface as `Unstable`

I think it would be good to use it for SASL too and that may
require the interface to be changed.

commit 70249b1502e32c0ea484e398b318545a500ecb12
Author: Ismael Juma 
Date:   2015-11-17T12:53:44Z

Mention in docs for `PRINCIPAL_BUILDER_CLASS_DOC` that it's only used for 
`SecurityProtocol.SSL`




> remove principal.builder.class from client configs
> --
>
> Key: KAFKA-2847
> URL: https://issues.apache.org/jira/browse/KAFKA-2847
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Ismael Juma
>
> Since only the broker needs to know the principal name, we can remove 
> principal.builder.class from both the producer and the consumer client config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2824) MiniKDC based tests don't run in VirtualBox

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009255#comment-15009255
 ] 

ASF GitHub Bot commented on KAFKA-2824:
---

GitHub user benstopford reopened a pull request:

https://github.com/apache/kafka/pull/520

KAFKA-2824: MiniKDC based tests don't run in VirtualBox

This is a hack which works. Is there a better way?

Build of the replication_test.py running here: 
http://jenkins.confluent.io/job/kafka_system_tests_branch_builder/174/console

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/benstopford/kafka fix-for-sasl-virtual-box

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #520


commit e398ae0af9020463fb3750d384fe00beb05c9ab2
Author: Ben Stopford 
Date:   2015-11-12T22:21:23Z

KAFKA-2824: force open binding in minikdc

commit 57fe6425b72cd0a977f3f8c9c46239f2feab9fef
Author: Ben Stopford 
Date:   2015-11-12T23:54:46Z

KAFKA-2824: add back in clean up

commit b1f2943e2ff10327120c06907b20f6f46510f8b7
Author: Ben Stopford 
Date:   2015-11-12T23:56:58Z

KAFKA-2824: tidy import




> MiniKDC based tests don't run in VirtualBox
> ---
>
> Key: KAFKA-2824
> URL: https://issues.apache.org/jira/browse/KAFKA-2824
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>Assignee: Ben Stopford
>
> When running system tests in virtualbox the miniKDC server isn't reachable. 
> Works fine in EC2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2801) Data read from network not processed by SSL transport layer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999683#comment-14999683
 ] 

ASF GitHub Bot commented on KAFKA-2801:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/493


> Data read from network not processed by SSL transport layer
> ---
>
> Key: KAFKA-2801
> URL: https://issues.apache.org/jira/browse/KAFKA-2801
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We have been seeing intermittent failures in our performance tests when 
> producer times out while waiting for metadata response, when running with 
> SSL. Digging deeper into this failure, this is a result of data that was read 
> into _SslTransporLayer.netReadBuffer_ during handshake processing, but never 
> processed later unless more data arrives on the network. In the case of the 
> producer, no more data is sent until the metadata response is received and 
> hence metadata request is never processed, leading to timeouts in the 
> producer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2803) Add hard bounce system test for distributed Kafka Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999682#comment-14999682
 ] 

ASF GitHub Bot commented on KAFKA-2803:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/494

KAFKA-2803: Add hard bounce system test for Kafka Connect.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2803-connect-hard-bounce-system-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/494.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #494


commit 584dd6a814c0e2a42f9b83459d958e4511cbf853
Author: Ewen Cheslack-Postava 
Date:   2015-11-10T17:06:57Z

KAFKA-2803: Add hard bounce system test for Kafka Connect.




> Add hard bounce system test for distributed Kafka Connect
> -
>
> Key: KAFKA-2803
> URL: https://issues.apache.org/jira/browse/KAFKA-2803
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>
> Similar to the existing clean bounce test, but use kill -9. The assertions we 
> can make are weaker here -- we can only check at least once delivery.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2770) Race condition causes Mirror Maker to hang during shutdown (new consumer)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999872#comment-14999872
 ] 

ASF GitHub Bot commented on KAFKA-2770:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/470


> Race condition causes Mirror Maker to hang during shutdown (new consumer)
> -
>
> Key: KAFKA-2770
> URL: https://issues.apache.org/jira/browse/KAFKA-2770
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> I recently added clean bounce with new consumer to the mirror maker tests 
> (https://github.com/apache/kafka/pull/427), and noticed that in this case the 
> mirror maker process (with new consumer) sometimes hangs and fails to stop 
> when stopped with kill -15
> {code:title=mirror_maker.log|borderStyle=solid}
> [2015-11-06 22:06:04,213] INFO Start clean shutdown. 
> (kafka.tools.MirrorMaker$)
> [2015-11-06 22:06:04,221] INFO Shutting down consumer threads. 
> (kafka.tools.MirrorMaker$)
> [2015-11-06 22:06:04,239] INFO [mirrormaker-thread-0] mirrormaker-thread-0 
> shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2015-11-06 22:06:04,253] INFO [mirrormaker-thread-0] Flushing producer. 
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2015-11-06 22:06:04,254] INFO [mirrormaker-thread-0] Committing consumer 
> offsets. (kafka.tools.MirrorMaker$MirrorMakerThread)
> Exception in thread "mirrormaker-thread-0" 
> org.apache.kafka.common.errors.WakeupException
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:304)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:194)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:184)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:154)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:347)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:895)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:869)
>   at 
> kafka.tools.MirrorMaker$MirrorMakerNewConsumer.commit(MirrorMaker.scala:522)
>   at kafka.tools.MirrorMaker$.commitOffsets(MirrorMaker.scala:338)
>   at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:406)
> [2015-11-06 22:06:29,448] DEBUG Connection with worker4/192.168.50.104 
> disconnected (org.apache.kafka.common.network.Selector)
> java.io.EOFException
>   at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:83)
>   at 
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
>   at 
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:160)
>   at 
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:141)
>   at org.apache.kafka.common.network.Selector.poll(Selector.java:288)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> The current working hypothesis is this:
> a WakeupException is being triggered during the finally block in mirror maker 
> by the call to commitOffsets, and the mirror maker thread dies before the 
> call to shutdownLatch.countDown(). Therefore the shutdownLatch.await() call 
> in awaitShutdown() blocks forever and the process never exits.
> Why can commitOffsets trigger a wakeup exception?
> The shutdown hook is triggered in another thread, and does this:
> shuttingDown = true
> mirrorMakerConsumer.stop()  # Calls consumer.wakeup()
> If the timing is right (wrong), the wakeup flag is set, but the mirrormaker 
> produce/consume loop exits without triggering the WakeupException, and the 
> WakeupException isn't thrown until commitOffsets() is called in the finally 
> block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2792) KafkaConsumer.close() can block unnecessarily due to leave group waiting for a reply

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997904#comment-14997904
 ] 

ASF GitHub Bot commented on KAFKA-2792:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/480

KAFKA-2792: Don't wait for a response to the leave group message when 
closing the new consumer.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2792-fix-blocking-consumer-close

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/480.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #480


commit a781fba82af6a6224f441885e22d3d6b35fc96f0
Author: Ewen Cheslack-Postava 
Date:   2015-11-10T02:56:08Z

KAFKA-2792: Don't wait for a response to the leave group message when 
closing the new consumer.




> KafkaConsumer.close() can block unnecessarily due to leave group waiting for 
> a reply
> 
>
> Key: KAFKA-2792
> URL: https://issues.apache.org/jira/browse/KAFKA-2792
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> The current implementation of close() waits for a response to LeaveGroup. 
> However, if we have an outstanding rebalance in the works, this can cause the 
> close() operation to have to wait for the entire rebalance process to 
> complete, which is annoying since the goal is to get rid of the consumer 
> object anyway. This is at best surprising and at worst can cause unexpected 
> bugs due to close() taking excessively long -- this was found due to 
> exceeding timeouts unexpectedly causing other operations in Kafka Connect to 
> timeout.
> Waiting for a response isn't necessary since as soon as the data is in the 
> TCP buffer, it'll be delivered to the broker. The client doesn't benefit at 
> all from seeing the close group. So we can instead just always send the 
> request 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2786) Only invoke SinkTask onPartitionsRevoked and commitOffsets after task has fully started

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997950#comment-14997950
 ] 

ASF GitHub Bot commented on KAFKA-2786:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/476


> Only invoke SinkTask onPartitionsRevoked and commitOffsets after task has 
> fully started
> ---
>
> Key: KAFKA-2786
> URL: https://issues.apache.org/jira/browse/KAFKA-2786
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
> Fix For: 0.9.0.0
>
>
> Currently we're invoking task.onPartitionsRevoked and commitOffsets on any 
> consumer onPartitionsRevoked callback. Since this happens even on the first 
> rebalance (which is somewhat unexpected given that partitions cannot actually 
> be revoked at that point), we will not be fully setup in the SinkTask and 
> this can throw exceptions.
> It turns out this is easy to miss for two reasons. First, the code is likely 
> to throw an exception, but the consumer catches this and handles it. We end 
> up not missing any important steps as a result. Second, the logging was a bit 
> screwed up and we weren't getting the full exception stacktrace, just the 
> toString(), so it was easy to miss that there was a problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2769) Integration tests for multi-consumer assignment including session timeouts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997453#comment-14997453
 ] 

ASF GitHub Bot commented on KAFKA-2769:
---

GitHub user apovzner opened a pull request:

https://github.com/apache/kafka/pull/472

KAFKA-2769:  Multi-consumer integration tests for consumer assignment incl. 
session timeouts and corresponding fixes

-- Refactored multi-consumer integration group assignment validation tests 
for round-robin assignment
-- Added multi-consumer integration tests for session timeout expiration:
   1. When a consumer stops polling
2. When a consumer calls close()
-- Fixes to issues found with session timeout expiration tests woth help 
from Jason Gustafson: Try to avoid  SendFailedException exception by cancelling 
the scheduled tasks and ensuring metadata update before sending group leave 
requests + send leave group request with retries.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apovzner/kafka cpkafka-81

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/472.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #472


commit 0dc57aa7bed768559da19707e672dfbb55c0460b
Author: Anna Povzner 
Date:   2015-11-06T04:02:55Z

KAFKA-2769: Multi-consumer integration tests for consumer group subsribe 
and timeouts and consumer timeout fixes.

-- Refactored multi-consumer integration group assignment validation tests 
for round-robin assignment
-- Added multi-consumer integration tests for session timeout expiration
-- Fixes to issues found with session timeout expiration tests woth help 
from Jason Gustafson.
1. shouldKeepMemberAlive(): when we are in the sync phase, we do want 
to expire members if we don't get any response
2. Try to avoid  SendFailedException exception by cancelling the 
scheduled tasks and ensuring metadata update before sending group leave 
requests.

commit 87554675c0ad5e51602c4cc37a40a9bb273b6dd0
Author: Anna Povzner 
Date:   2015-11-07T00:10:04Z

KAFKA-2769: More reliable sending of leave group request in the consumer on 
consumer close().

commit 353ef1c079f39ad15a91ba26f64f95c541d517f1
Author: Anna Povzner 
Date:   2015-11-09T19:56:36Z

Reverted a change in shouldKeepMemberAlive() to check for sync callback set

commit 83f703d6cb8720ac12e25692ce273e15813e89f6
Author: Anna Povzner 
Date:   2015-11-09T20:52:15Z

fixed minor test issues




> Integration tests for multi-consumer assignment including session timeouts
> --
>
> Key: KAFKA-2769
> URL: https://issues.apache.org/jira/browse/KAFKA-2769
> Project: Kafka
>  Issue Type: Test
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> We currently don't have integration tests for multi-consumer range assignment 
> and session timeout expiration scenarios.
> This card is to add multi-consumer integration tests for range assignment:
> -- initial assignment 
> -- adding a new consumer to an existing group
> -- adding and removing a topic to/from subscription
> -- session timeout expiration when consumer stops polling and on consumer 
> close().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2785) Copycat jars not included in release tar gz

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997586#comment-14997586
 ] 

ASF GitHub Bot commented on KAFKA-2785:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/473

KAFKA-2785: Include Kafka Connect jars in releaseTarGz.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2785-include-copycat-jars-in-release

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/473.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #473


commit 035614d4430c0dad7d392abd2ff2dd0d9cd75b35
Author: Ewen Cheslack-Postava 
Date:   2015-11-09T22:57:19Z

KAFKA-2785: Include Kafka Connect jars in releaseTarGz.




> Copycat jars not included in release tar gz
> ---
>
> Key: KAFKA-2785
> URL: https://issues.apache.org/jira/browse/KAFKA-2785
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> Currently they are missing because they are separate jars but only 
> dependencies of core are pulled in unless explicitly added to the tar gz.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2785) Copycat jars not included in release tar gz

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997605#comment-14997605
 ] 

ASF GitHub Bot commented on KAFKA-2785:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/473


> Copycat jars not included in release tar gz
> ---
>
> Key: KAFKA-2785
> URL: https://issues.apache.org/jira/browse/KAFKA-2785
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> Currently they are missing because they are separate jars but only 
> dependencies of core are pulled in unless explicitly added to the tar gz.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2773) Vagrant provision fails if num_brokers or num_zookeepers is nonzero

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997749#comment-14997749
 ] 

ASF GitHub Bot commented on KAFKA-2773:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/454


> Vagrant provision fails if num_brokers or num_zookeepers is nonzero
> ---
>
> Key: KAFKA-2773
> URL: https://issues.apache.org/jira/browse/KAFKA-2773
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>Priority: Trivial
> Fix For: 0.9.0.0
>
>
> Changes to vagrant provisioning for kafka system tests updated the default 
> path from "kafka" to "kafka-trunk" on the vagrant virtual machines.
> We neglected to update the corresponding path in vagrant/broker.sh and 
> vagrant/zk.sh. Therefore provisioning a static kafka cluster with Vagrant 
> currently fails.
> The fix here is just to update the corresponding path in vagrant/broker.sh 
> and vagrant/zk.sh



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2786) Only invoke SinkTask onPartitionsRevoked and commitOffsets after task has fully started

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997669#comment-14997669
 ] 

ASF GitHub Bot commented on KAFKA-2786:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/476

KAFKA-2786: Only respond to SinkTask onPartitionsRevoked after the 
WorkerSinkTask has finished starting up.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2786-on-partitions-assigned-only-after-start

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/476.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #476






> Only invoke SinkTask onPartitionsRevoked and commitOffsets after task has 
> fully started
> ---
>
> Key: KAFKA-2786
> URL: https://issues.apache.org/jira/browse/KAFKA-2786
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
> Fix For: 0.9.0.0
>
>
> Currently we're invoking task.onPartitionsRevoked and commitOffsets on any 
> consumer onPartitionsRevoked callback. Since this happens even on the first 
> rebalance (which is somewhat unexpected given that partitions cannot actually 
> be revoked at that point), we will not be fully setup in the SinkTask and 
> this can throw exceptions.
> It turns out this is easy to miss for two reasons. First, the code is likely 
> to throw an exception, but the consumer catches this and handles it. We end 
> up not missing any important steps as a result. Second, the logging was a bit 
> screwed up and we weren't getting the full exception stacktrace, just the 
> toString(), so it was easy to miss that there was a problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2787) Refactor gradle build

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997683#comment-14997683
 ] 

ASF GitHub Bot commented on KAFKA-2787:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/477

KAFKA-2787: Refactor gradle build

This is a first pass. I stopped here given its already a decent amount of 
change. 
I will follow up with more in another pull request if this is received well.

What was done:
- Move gradle "script plugins" into /gradle folder
- Update rat plugin grgit dep (can update since java 1.6 support was 
dropped)
- De-duplicate common test deps and configs by moving to subprojects config
- De-duplicate common Checkstyle deps by moving to subprojects config
- Fix Checkstyle on java in core
- Centralize dependencies/versions in dependencies.gradle
- Setup Scala version dependencies to be more flexible (could support 2.12)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka gradle-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #477


commit 3e63aa15d6377821611e29b34654f37c0fb6126f
Author: Grant Henke 
Date:   2015-11-09T23:46:48Z

KAFKA-2787: Refactor gradle build




> Refactor gradle build
> -
>
> Key: KAFKA-2787
> URL: https://issues.apache.org/jira/browse/KAFKA-2787
> Project: Kafka
>  Issue Type: Task
>  Components: build
>Affects Versions: 0.8.2.2
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.0.0
>
>
> The build files are quite large with a lot of duplication and overlap. This 
> could lead to mistakes, reduce readability and functionality, and hinder 
> future changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2784) Mirror maker should swallow exceptions during shutdown.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997696#comment-14997696
 ] 

ASF GitHub Bot commented on KAFKA-2784:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/478

KAFKA-2784: swallow exceptions when mirror maker exits.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2784

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/478.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #478


commit 7272a4488780529cb0dc6a9e1abc14a6f8117cbe
Author: Jiangjie Qin 
Date:   2015-11-10T00:01:13Z

KAFKA-2784: swallow exceptions when mirror maker exits.




> Mirror maker should swallow exceptions during shutdown.
> ---
>
> Key: KAFKA-2784
> URL: https://issues.apache.org/jira/browse/KAFKA-2784
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>
> Mirror maker should swallow exceptions during shutdown to make sure shutdown 
> latch is pulled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2379) Add Copycat documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997733#comment-14997733
 ] 

ASF GitHub Bot commented on KAFKA-2379:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/475


> Add Copycat documentation
> -
>
> Key: KAFKA-2379
> URL: https://issues.apache.org/jira/browse/KAFKA-2379
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Starting this out pretty broad as it can cover a lot. Some ideas:
> * Normal intro/readme type stuff
> * User guide - how to run in standalone/distributed mode. Connector/tasks 
> concepts and what they mean in practice. Fault tolerance & offsets. REST 
> interface, Copycat as a service, etc.
> * Dev guide - connectors/partitions/records/offsets/tasks. All the APIs, 
> specific examples for implementing APIs, resuming from previous offsets, 
> dynamic sets of partitions, how to work with the runtime data API, etc.
> * System design - KIP-26 + more - why we ended up on the design we did, 
> comparisons to other systems w/ low level details, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2379) Add Copycat documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997656#comment-14997656
 ] 

ASF GitHub Bot commented on KAFKA-2379:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/475

KAFKA-2379: Add basic documentation for Kafka Connect.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka kafka-2379-connect-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/475.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #475


commit 3a6da2cbc6d8524cec04cffddd3db53967d26073
Author: Ewen Cheslack-Postava 
Date:   2015-11-09T23:34:20Z

KAFKA-2379: Add basic documentation for Kafka Connect.




> Add Copycat documentation
> -
>
> Key: KAFKA-2379
> URL: https://issues.apache.org/jira/browse/KAFKA-2379
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Starting this out pretty broad as it can cover a lot. Some ideas:
> * Normal intro/readme type stuff
> * User guide - how to run in standalone/distributed mode. Connector/tasks 
> concepts and what they mean in practice. Fault tolerance & offsets. REST 
> interface, Copycat as a service, etc.
> * Dev guide - connectors/partitions/records/offsets/tasks. All the APIs, 
> specific examples for implementing APIs, resuming from previous offsets, 
> dynamic sets of partitions, how to work with the runtime data API, etc.
> * System design - KIP-26 + more - why we ended up on the design we did, 
> comparisons to other systems w/ low level details, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2769) Integration tests for multi-consumer assignment including session timeouts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14997764#comment-14997764
 ] 

ASF GitHub Bot commented on KAFKA-2769:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/472


> Integration tests for multi-consumer assignment including session timeouts
> --
>
> Key: KAFKA-2769
> URL: https://issues.apache.org/jira/browse/KAFKA-2769
> Project: Kafka
>  Issue Type: Test
>Reporter: Anna Povzner
>Assignee: Anna Povzner
> Fix For: 0.9.0.0
>
>
> We currently don't have integration tests for multi-consumer range assignment 
> and session timeout expiration scenarios.
> This card is to add multi-consumer integration tests for range assignment:
> -- initial assignment 
> -- adding a new consumer to an existing group
> -- adding and removing a topic to/from subscription
> -- session timeout expiration when consumer stops polling and on consumer 
> close().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2687) Allow GroupMetadataRequest to return member metadata when received by group coordinator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982035#comment-14982035
 ] 

ASF GitHub Bot commented on KAFKA-2687:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/388

KAFKA-2687: Add support for ListGroups and DescribeGroup APIs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka K2687

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/388.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #388


commit f5f9a9159bd34b0ebb5fa1ed818faea2a5a3c44c
Author: Jason Gustafson 
Date:   2015-10-28T22:34:04Z

KAFKA-2687: Add support for ListGroups and DescribeGroup APIs




> Allow GroupMetadataRequest to return member metadata when received by group 
> coordinator
> ---
>
> Key: KAFKA-2687
> URL: https://issues.apache.org/jira/browse/KAFKA-2687
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Since the new consumer currently has no persistence in Zookeeper (pending 
> outcome of KAFKA-2017), there is no way for administrators to investigate 
> group status including getting the list of members in the group and their 
> partition assignments. We therefore propose to modify GroupMetadataRequest 
> (previously known as ConsumerMetadataRequest) to return group metadata when 
> received by the respective group's coordinator. When received by another 
> broker, the request will be handled as before: by only returning coordinator 
> host and port information.
> {code}
> GroupMetadataRequest => GroupId IncludeMetadata
>   GroupId => String
>   IncludeMetadata => Boolean
> GroupMetadataResponse => ErrorCode Coordinator GroupMetadata
>   ErrorCode => int16
>   Coordinator => Id Host Port
> Id => int32
> Host => string
> Port => int32
>   GroupMetadata => State ProtocolType Generation Protocol Leader  Members
> State => String
> ProtocolType => String
> Generation => int32
> Protocol => String
> Leader => String
> Members => [Member MemberMetadata MemberAssignment]
>   Member => MemberIp ClientId
> MemberIp => String
> ClientId => String
>   MemberMetadata => Bytes
>   MemberAssignment => Bytes
> {code}
> The request schema includes a flag to indicate whether metadata is needed, 
> which saves clients from having to read all group metadata when they are just 
> trying to find the coordinator. This is important to reduce group overhead 
> for use cases which involve a large number of topic subscriptions (e.g. 
> mirror maker).
> Tools will use the protocol type to determine how to parse metadata. For 
> example, when the protocolType is "consumer", the tool can use 
> ConsumerProtocol to parse the member metadata as topic subscriptions and 
> partition assignments. 
> The detailed proposal can be found below.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-40%3A+ListGroups+and+DescribeGroup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984326#comment-14984326
 ] 

ASF GitHub Bot commented on KAFKA-2718:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/399

KAFKA-2718: Avoid reusing temporary directories in core unit tests

Retry to find new directory and cleanup on exit.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2718

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #399


commit a56cc3fa9c0ccf2e1b3556d3e47d5dc36ae2c7b6
Author: Rajini Sivaram 
Date:   2015-11-01T09:58:36Z

KAFKA-2718: Avoid reusing temporary directories in core unit tests




> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2719) Kafka classpath has grown too large and breaks some system tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984392#comment-14984392
 ] 

ASF GitHub Bot commented on KAFKA-2719:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/400

KAFKA-2719: Use wildcard classpath for dependant-libs

PR switches to wildcard classpath for dependant libs to restrict the length 
of classpath, thereby reducing command line length.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2719

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #400


commit f49a366334391074456859ac42059c0752bc40d2
Author: Rajini Sivaram 
Date:   2015-11-01T12:40:23Z

KAFKA-2719: Use wildcard classpath for dependant-libs to reduce command 
line length




> Kafka classpath has grown too large and breaks some system tests
> 
>
> Key: KAFKA-2719
> URL: https://issues.apache.org/jira/browse/KAFKA-2719
> Project: Kafka
>  Issue Type: Bug
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> The jars added under KAFKA-2369 makes the Kafka command line used in system 
> tests much higher than 4096 due to more jars in the classpath. Since the ps 
> command used to find processes in system tests truncates the command line, 
> some system tests are failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2681) SASL authentication in official docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985669#comment-14985669
 ] 

ASF GitHub Bot commented on KAFKA-2681:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/401


> SASL authentication in official docs
> 
>
> Key: KAFKA-2681
> URL: https://issues.apache.org/jira/browse/KAFKA-2681
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SASL 
> authentication:
> http://kafka.apache.org/documentation.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2722) Improve ISR change propagation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985686#comment-14985686
 ] 

ASF GitHub Bot commented on KAFKA-2722:
---

Github user becketqin closed the pull request at:

https://github.com/apache/kafka/pull/402


> Improve ISR change propagation
> --
>
> Key: KAFKA-2722
> URL: https://issues.apache.org/jira/browse/KAFKA-2722
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Currently the ISR change propagation interval is hard coded to 5 seconds, 
> this might still create a lot of ISR change propagation for a large cluster 
> in cases such as rolling bounce. The patch uses a dynamic propagation 
> interval and fixed a performance bug in IsrChangeNotificationListener on 
> controller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2722) Improve ISR change propagation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985687#comment-14985687
 ] 

ASF GitHub Bot commented on KAFKA-2722:
---

GitHub user becketqin reopened a pull request:

https://github.com/apache/kafka/pull/402

KAFKA-2722: Improve ISR change propagation.

The patch has two changes:
1. fixed a bug in controller that it sends UpdateMetadataRequest of all the 
partitions in the cluster.
2. Uses the following rules to propagate ISR change: 1) if there are ISR 
changes pending propagation and the last ISR change is more than five seconds 
ago, propagate the changes. 2) if there is ISR change at T in the recent five 
seconds, delay the propagation until T + 5s. 3) if the last propagation is more 
than 1 min ago, ignore rule No.2 and propagate ISR change if there are changes 
pending propagation.

This algorithm avoids a fixed configuration of ISR propagation interval as 
we discussed about in KIP-29.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2722

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #402


commit 13892856d806183536657f0c3ea2aa63b1f1c4f2
Author: Jiangjie Qin 
Date:   2015-11-02T01:26:27Z

Improve ISR change propagation.




> Improve ISR change propagation
> --
>
> Key: KAFKA-2722
> URL: https://issues.apache.org/jira/browse/KAFKA-2722
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Currently the ISR change propagation interval is hard coded to 5 seconds, 
> this might still create a lot of ISR change propagation for a large cluster 
> in cases such as rolling bounce. The patch uses a dynamic propagation 
> interval and fixed a performance bug in IsrChangeNotificationListener on 
> controller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2716) Make Kafka core not depend on log4j-appender

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985754#comment-14985754
 ] 

ASF GitHub Bot commented on KAFKA-2716:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/405

KAFKA-2716: Make Kafka core not depend on log4j-appender



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2716

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #405


commit d4d69ab4fec81efd18eee68303291bb90dc4ca29
Author: Ashish Singh 
Date:   2015-11-02T18:54:28Z

KAFKA-2716: Make Kafka core not depend on log4j-appender




> Make Kafka core not depend on log4j-appender
> 
>
> Key: KAFKA-2716
> URL: https://issues.apache.org/jira/browse/KAFKA-2716
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Investigate why core needs to depend on log4j-appender. AFAIK, there is no 
> real dependency, however it the dependency is removed, tests won't build it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2017) Persist Coordinator State for Coordinator Failover

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986850#comment-14986850
 ] 

ASF GitHub Bot commented on KAFKA-2017:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/386


> Persist Coordinator State for Coordinator Failover
> --
>
> Key: KAFKA-2017
> URL: https://issues.apache.org/jira/browse/KAFKA-2017
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2017.patch, KAFKA-2017_2015-05-20_09:13:39.patch, 
> KAFKA-2017_2015-05-21_19:02:47.patch
>
>
> When a coordinator fails, the group membership protocol tries to failover to 
> a new coordinator without forcing all the consumers rejoin their groups. This 
> is possible if the coordinator persists its state so that the state can be 
> transferred during coordinator failover. This state consists of most of the 
> information in GroupRegistry and ConsumerRegistry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2518) Update NOTICE file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985792#comment-14985792
 ] 

ASF GitHub Bot commented on KAFKA-2518:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/404


> Update NOTICE file
> --
>
> Key: KAFKA-2518
> URL: https://issues.apache.org/jira/browse/KAFKA-2518
> Project: Kafka
>  Issue Type: Bug
>  Components: packaging
>Reporter: Flavio Junqueira
>Assignee: Gwen Shapira
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> According to this page from ASF legal:
> {noformat}
> http://www.apache.org/legal/src-headers.html
> {noformat}
> the years in the NOTICE header should reflect the product name and years of 
> distribution of the current and past versions of the product. The current 
> NOTICE file says only 2012. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2441) SSL/TLS in official docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985779#comment-14985779
 ] 

ASF GitHub Bot commented on KAFKA-2441:
---

GitHub user gwenshap opened a pull request:

https://github.com/apache/kafka/pull/406

KAFKA-2441: SSL/TLS in official docs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2441

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #406


commit f5dcf2fb69a44e3083973af362b2c337bcc48ff3
Author: Gwen Shapira 
Date:   2015-11-02T19:01:03Z

KAFKA-2441L: SSL/TLS in official docs




> SSL/TLS in official docs
> 
>
> Key: KAFKA-2441
> URL: https://issues.apache.org/jira/browse/KAFKA-2441
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SSL/TLS:
> http://kafka.apache.org/documentation.html
> There is already a wiki page where some of the information is already present:
> https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2726) ntpdate causes vagrant provision to fail if ntp running

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985998#comment-14985998
 ] 

ASF GitHub Bot commented on KAFKA-2726:
---

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/407

KAFKA-2726: Fix port collision between ntpdate and ntp daemon

@gwenshap Can you take a quick look? I have verified the change allows 
successful `vagrant provision` even with ntp daemon already running on the vm.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
KAFKA-2726-ntp-port-collision

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #407


commit 4600ea1fa4a6dfe43b639210e31dae325413907d
Author: Geoff Anderson 
Date:   2015-11-02T20:42:26Z

Fix port collision between ntpdate and ntp daemon




> ntpdate causes vagrant provision to fail if ntp running
> ---
>
> Key: KAFKA-2726
> URL: https://issues.apache.org/jira/browse/KAFKA-2726
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> If ntp daemon is already running, vagrant provision can fail because of port 
> collision



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2715) Remove the previous system test folder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983469#comment-14983469
 ] 

ASF GitHub Bot commented on KAFKA-2715:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/392


> Remove the previous system test folder
> --
>
> Key: KAFKA-2715
> URL: https://issues.apache.org/jira/browse/KAFKA-2715
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>  Labels: test
> Fix For: 0.9.0.0
>
>
> As part of KAFKA-25, we want to remove the existing system tests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2691) Improve handling of authorization failure during metadata refresh

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983605#comment-14983605
 ] 

ASF GitHub Bot commented on KAFKA-2691:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/394

KAFKA-2691: Improve handling of authorization failure during metadata 
refresh



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2691

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #394


commit 722980095636f8d0f633aa1f40a6aa5736facdbe
Author: Jason Gustafson 
Date:   2015-10-30T23:40:00Z

KAFKA-2691: Improve handling of authorization failure during metadata 
refresh




> Improve handling of authorization failure during metadata refresh
> -
>
> Key: KAFKA-2691
> URL: https://issues.apache.org/jira/browse/KAFKA-2691
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Jason Gustafson
> Fix For: 0.9.0.0
>
>
> There are two problems, one more severe than the other:
> 1. The consumer blocks indefinitely if there is non-transient authorization 
> failure during metadata refresh due to KAFKA-2391
> 2. We get a TimeoutException instead of an AuthorizationException in the 
> producer for the same case
> If the fix for KAFKA-2391 is to add a timeout, then we will have issue `2` in 
> both producer and consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2714) Add integration tests for exceptional cases in Fetching for new consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983409#comment-14983409
 ] 

ASF GitHub Bot commented on KAFKA-2714:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/393


> Add integration tests for exceptional cases in Fetching for new consumer
> 
>
> Key: KAFKA-2714
> URL: https://issues.apache.org/jira/browse/KAFKA-2714
> Project: Kafka
>  Issue Type: Test
>Reporter: Anna Povzner
>Assignee: Anna Povzner
> Fix For: 0.9.0.0
>
>
> We currently don't have integration tests for exceptional cases in fetches 
> for new consumer. This ticket is to create the following test scenarios:
> 1. When reset policy is NONE, verify that NoOffsetForPartitionException is 
> thrown if no initial position is set.
> 2. When reset policy is NONE, verify that OffsetOutOfRange is thrown if you 
> seek out of range.
> 3. Verify RecordTooLargeException is thrown if a message is too large for the 
> configured fetch size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2714) Add integration tests for exceptional cases in Fetching for new consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983358#comment-14983358
 ] 

ASF GitHub Bot commented on KAFKA-2714:
---

GitHub user apovzner opened a pull request:

https://github.com/apache/kafka/pull/393

KAFKA-2714: Added integration tests for exceptional cases in fetching



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apovzner/kafka cpkafka-84

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #393


commit 4f175d813270ae2943dd4466c51bedbf11e819ea
Author: Anna Povzner 
Date:   2015-10-29T22:21:01Z

MINOR: Added integration tests for exceptional cases in fetching

commit 6aa6aaf4b4f3ebb4f101fc0a5088195d52a2a0ba
Author: Anna Povzner 
Date:   2015-10-30T00:00:50Z

MINOR: Checking correct values in exceptions thrown in integration tests 
for exceptional cases in fetching




> Add integration tests for exceptional cases in Fetching for new consumer
> 
>
> Key: KAFKA-2714
> URL: https://issues.apache.org/jira/browse/KAFKA-2714
> Project: Kafka
>  Issue Type: Test
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> We currently don't have integration tests for exceptional cases in fetches 
> for new consumer. This ticket is to create the following test scenarios:
> 1. When reset policy is NONE, verify that NoOffsetForPartitionException is 
> thrown if no initial position is set.
> 2. When reset policy is NONE, verify that OffsetOutOfRange is thrown if you 
> seek out of range.
> 3. Verify RecordTooLargeException is thrown if a message is too large for the 
> configured fetch size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2714) Add integration tests for exceptional cases in Fetching for new consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983348#comment-14983348
 ] 

ASF GitHub Bot commented on KAFKA-2714:
---

Github user apovzner closed the pull request at:

https://github.com/apache/kafka/pull/384


> Add integration tests for exceptional cases in Fetching for new consumer
> 
>
> Key: KAFKA-2714
> URL: https://issues.apache.org/jira/browse/KAFKA-2714
> Project: Kafka
>  Issue Type: Test
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> We currently don't have integration tests for exceptional cases in fetches 
> for new consumer. This ticket is to create the following test scenarios:
> 1. When reset policy is NONE, verify that NoOffsetForPartitionException is 
> thrown if no initial position is set.
> 2. When reset policy is NONE, verify that OffsetOutOfRange is thrown if you 
> seek out of range.
> 3. Verify RecordTooLargeException is thrown if a message is too large for the 
> configured fetch size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2369) Add Copycat REST API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983430#comment-14983430
 ] 

ASF GitHub Bot commented on KAFKA-2369:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/378


> Add Copycat REST API
> 
>
> Key: KAFKA-2369
> URL: https://issues.apache.org/jira/browse/KAFKA-2369
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> Add a REST API for Copycat. At a minimum, for a single worker this should 
> support:
> * add/remove connector
> * connector status
> * task status
> * worker status
> In distributed mode this should handle forwarding if necessary, but it may 
> make sense to defer the distributed support for a later JIRA.
> This will require the addition of new dependencies to support implementing 
> the REST API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2711) SaslClientAuthenticator no longer needs KerberosNameParser in constructor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982384#comment-14982384
 ] 

ASF GitHub Bot commented on KAFKA-2711:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/390

KAFKA-2711; SaslClientAuthenticator no longer needs KerberosNameParser in 
constructor

Also refactor `KerberosNameParser` and `KerberosName` to make the code
clearer and easier to use when `shortName` is not needed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka kafka-2711

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/390.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #390


commit 751e52ffee04401a6bf8170469594fe5f526bff1
Author: Ismael Juma 
Date:   2015-10-30T11:13:11Z

Remove unnecessary usage of `KerberosNameParser` in 
`SaslClientAuthenticator`

Also refactor `KerberosNameParser` and `KerberosName` to make the code
clearer.




> SaslClientAuthenticator no longer needs KerberosNameParser in constructor
> -
>
> Key: KAFKA-2711
> URL: https://issues.apache.org/jira/browse/KAFKA-2711
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Priority: Minor
> Fix For: 0.9.1
>
>
> Since the sasl client doesn't need to know the principal name, we don't need 
> to pass in KerberosNameParser to SaslClientAuthenticator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2698) add paused API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984633#comment-14984633
 ] 

ASF GitHub Bot commented on KAFKA-2698:
---

GitHub user thomaslee opened a pull request:

https://github.com/apache/kafka/pull/403

KAFKA-2698: Add paused() method to o.a.k.c.c.Consumer

As per KAFKA-2698, this adds a `paused()` method to the Consumer interface 
such that client code can query Consumer implementations for paused partitions.

Somewhat new to the code base but I understand this may require a KIP given 
this changes APIs: is this required even for backward-compatible changes like 
this?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thomaslee/kafka tom_consumer_paused_query

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #403


commit ec92932395c401a8f87986bd2e91e296500c3612
Author: Tom Lee 
Date:   2015-11-02T00:58:38Z

KAFKA-2698: Add paused() method to o.a.k.c.c.Consumer




> add paused API
> --
>
> Key: KAFKA-2698
> URL: https://issues.apache.org/jira/browse/KAFKA-2698
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
> Fix For: 0.9.0.0
>
>
> org.apache.kafka.clients.consumer.Consumer tends to follow a pattern of 
> having an action API paired with a query API:
> subscribe() has subscription()
> assign() has assignment()
> There's no analogous API for pause.
> Should there be a paused() API returning Set?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2681) SASL authentication in official docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984607#comment-14984607
 ] 

ASF GitHub Bot commented on KAFKA-2681:
---

Github user gwenshap closed the pull request at:

https://github.com/apache/kafka/pull/401


> SASL authentication in official docs
> 
>
> Key: KAFKA-2681
> URL: https://issues.apache.org/jira/browse/KAFKA-2681
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SASL 
> authentication:
> http://kafka.apache.org/documentation.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2681) SASL authentication in official docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984608#comment-14984608
 ] 

ASF GitHub Bot commented on KAFKA-2681:
---

GitHub user gwenshap reopened a pull request:

https://github.com/apache/kafka/pull/401

KAFKA-2681: Added SASL documentation. It doesn't look great, but contains 
all the…

… info from the wiki

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2681

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #401


commit ac9ea2e8bf5e25d4ab5fca8f72107dcd440b8933
Author: Gwen Shapira 
Date:   2015-11-02T00:18:34Z

Added SASL documentation. It doesn't look great, but contains all the info 
from the wiki




> SASL authentication in official docs
> 
>
> Key: KAFKA-2681
> URL: https://issues.apache.org/jira/browse/KAFKA-2681
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SASL 
> authentication:
> http://kafka.apache.org/documentation.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2574) Add ducktape based ssl test for KafkaLog4jAppender

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983706#comment-14983706
 ] 

ASF GitHub Bot commented on KAFKA-2574:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/319


> Add ducktape based ssl test for KafkaLog4jAppender
> --
>
> Key: KAFKA-2574
> URL: https://issues.apache.org/jira/browse/KAFKA-2574
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.9.0.0
>
>
> KAFKA-2447 adds support for using SSL in KafkaLog4jAppender and KAFKA-2531 
> adds basic ducktape based tests for KafkaLog4jAppender. This should add 
> ducktape based ssl tests for KafkaLog4jAppender.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2717) Add kafka logbak appender

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983932#comment-14983932
 ] 

ASF GitHub Bot commented on KAFKA-2717:
---

GitHub user vesense opened a pull request:

https://github.com/apache/kafka/pull/398

KAFKA-2717: Add kafka logbak appender

https://issues.apache.org/jira/browse/KAFKA-2717

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/kafka kafka-logback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #398


commit 8403829dd194ad99f507acc94a01274a6d2b5f39
Author: vesense 
Date:   2015-10-30T08:34:21Z

ignore subproject .gitignore file

commit eb7d7d512f2087f2914c6e538b242c2c6d98d5d3
Author: vesense 
Date:   2015-10-30T13:14:06Z

add logback appender

commit 9de9c4ed45c2b3fc4263c2b0d1dd14b2f40fbb12
Author: vesense 
Date:   2015-10-31T10:44:59Z

Add kafka logbak appender




> Add kafka logbak appender
> -
>
> Key: KAFKA-2717
> URL: https://issues.apache.org/jira/browse/KAFKA-2717
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Xin Wang
>
> Since many applications use logback as their log framework.
> So, KafkaLogbakAppender would make it easier for integrating with kafka, just 
> like KafkaLog4jAppender.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2697) add leave group logic to the consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990643#comment-14990643
 ] 

ASF GitHub Bot commented on KAFKA-2697:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/414


> add leave group logic to the consumer
> -
>
> Key: KAFKA-2697
> URL: https://issues.apache.org/jira/browse/KAFKA-2697
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Jason Gustafson
> Fix For: 0.9.0.0
>
>
> KAFKA-2397 added logic on the coordinator to handle LeaveGroupRequests. We 
> need to add logic to KafkaConsumer to send out a LeaveGroupRequest on close.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2258) Port mirrormaker_testsuite

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990928#comment-14990928
 ] 

ASF GitHub Bot commented on KAFKA-2258:
---

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/427

KAFKA-2258[WIP]: add failover to mirrormaker test

This PR adds failover to simple end to end mirror maker test

Marked as WIP for 2 reasons:
- We may want to add a couple more test cases where kafka is being used to 
store offsets
- There appears to be a test failure in the hard failover case

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka KAFKA-2258-mirrormaker-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #427


commit b5db56912720e98ab3f191370a6dadca4efed1f9
Author: Geoff Anderson 
Date:   2015-10-28T23:59:51Z

Added sketch of mirror maker failover

commit 1c9bc561bf5047360974cfb121f039c63a2bf1a8
Author: Geoff Anderson 
Date:   2015-11-05T01:41:46Z

Cleaned up logging

commit 62cfe5db519975f74e4ccf5ff5695b36322ae6cb
Author: Geoff Anderson 
Date:   2015-11-05T01:44:02Z

Removed extra spaces from producer.properties




> Port mirrormaker_testsuite
> --
>
> Key: KAFKA-2258
> URL: https://issues.apache.org/jira/browse/KAFKA-2258
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.9.0.0
>
>
> Port mirrormaker_testsuite to run on ducktape



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2745) Update JavaDoc for the new / updated APIs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990700#comment-14990700
 ] 

ASF GitHub Bot commented on KAFKA-2745:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/425

KAFKA-2745: Update JavaDoc for new / updated consumer APIs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2745

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #425


commit 6a9b18127ebd26cab5ab8b2a66bc6a296e982c50
Author: Guozhang Wang 
Date:   2015-11-04T23:06:38Z

header docs

commit 297842087979a893de5447055d5fe89a00bae1d4
Author: Guozhang Wang 
Date:   2015-11-04T23:06:49Z

Merge branch 'trunk' of https://github.com/apache/kafka into K2745

commit d4855042eb29eb90ab05246ec4f06f4c3d4325e3
Author: Guozhang Wang 
Date:   2015-11-04T23:27:07Z

Function java doc




> Update JavaDoc for the new / updated APIs
> -
>
> Key: KAFKA-2745
> URL: https://issues.apache.org/jira/browse/KAFKA-2745
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2727) initialize only the part of the topology relevant to the task

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987742#comment-14987742
 ] 

ASF GitHub Bot commented on KAFKA-2727:
---

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/411

KAFKA-2727: Topology partial construction

@guozhangwang 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka topology_partial_construction

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/411.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #411


commit e561254d9f3e6927a9e0d62c85d7144d936d678b
Author: Yasuhiro Matsuda 
Date:   2015-10-29T17:03:10Z

partial construction of topology

commit 3ae930c43e0f2f60caa183d7265d3e69442a3d96
Author: Yasuhiro Matsuda 
Date:   2015-11-02T22:59:42Z

Merge branch 'trunk' of github.com:apache/kafka into 
topology_partial_construction

commit f984b3214e72c678ea6f45a564325eb212c4ccdf
Author: Yasuhiro Matsuda 
Date:   2015-11-02T23:13:31Z

cleanup

commit d170007d0df57af3afbe22c121aac0bd10dbeb7e
Author: Yasuhiro Matsuda 
Date:   2015-11-02T23:24:06Z

test




> initialize only the part of the topology relevant to the task
> -
>
> Key: KAFKA-2727
> URL: https://issues.apache.org/jira/browse/KAFKA-2727
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.0
>Reporter: Yasuhiro Matsuda
>Assignee: Yasuhiro Matsuda
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2716) Make Kafka core not depend on log4j-appender

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987800#comment-14987800
 ] 

ASF GitHub Bot commented on KAFKA-2716:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/405


> Make Kafka core not depend on log4j-appender
> 
>
> Key: KAFKA-2716
> URL: https://issues.apache.org/jira/browse/KAFKA-2716
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.9.0.0
>
>
> Investigate why core needs to depend on log4j-appender. AFAIK, there is no 
> real dependency, however it the dependency is removed, tests won't build it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2724) Document ZooKeeper authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987709#comment-14987709
 ] 

ASF GitHub Bot commented on KAFKA-2724:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/409


> Document ZooKeeper authentication 
> --
>
> Key: KAFKA-2724
> URL: https://issues.apache.org/jira/browse/KAFKA-2724
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Add documentation for ZooKeeper authentication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2441) SSL/TLS in official docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988278#comment-14988278
 ] 

ASF GitHub Bot commented on KAFKA-2441:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/406


> SSL/TLS in official docs
> 
>
> Key: KAFKA-2441
> URL: https://issues.apache.org/jira/browse/KAFKA-2441
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SSL/TLS:
> http://kafka.apache.org/documentation.html
> There is already a wiki page where some of the information is already present:
> https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2734) kafka-console-consumer throws NoSuchElementException on not specifying topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988335#comment-14988335
 ] 

ASF GitHub Bot commented on KAFKA-2734:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/412

KAFKA-2734: kafka-console-consumer throws NoSuchElementException on n…

…ot specifying topic

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2734

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #412


commit 1b8522773d2ff799bb2228c5003d8fce5dcd4e86
Author: Ashish Singh 
Date:   2015-11-03T22:42:21Z

KAFKA-2734: kafka-console-consumer throws NoSuchElementException on not 
specifying topic




> kafka-console-consumer throws NoSuchElementException on not specifying topic
> 
>
> Key: KAFKA-2734
> URL: https://issues.apache.org/jira/browse/KAFKA-2734
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> The logic of argument checking is flawed for kafka-console-consumer. Throws 
> below mentioned exception when topic is not specified. Users wont have a clue 
> what went wrong.
> {code}
> Exception in thread "main" java.util.NoSuchElementException: head of empty 
> list
>   at scala.collection.immutable.Nil$.head(List.scala:337)
>   at scala.collection.immutable.Nil$.head(List.scala:334)
>   at 
> kafka.tools.ConsoleConsumer$ConsumerConfig.(ConsoleConsumer.scala:244)
>   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:40)
>   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2687) Add support for ListGroups and DescribeGroup APIs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988327#comment-14988327
 ] 

ASF GitHub Bot commented on KAFKA-2687:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/388


> Add support for ListGroups and DescribeGroup APIs
> -
>
> Key: KAFKA-2687
> URL: https://issues.apache.org/jira/browse/KAFKA-2687
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Since the new consumer currently has no persistence in Zookeeper (pending 
> outcome of KAFKA-2017), there is no way for administrators to investigate 
> group status including getting the list of members in the group and their 
> partition assignments. We therefore propose to modify GroupMetadataRequest 
> (previously known as ConsumerMetadataRequest) to return group metadata when 
> received by the respective group's coordinator. When received by another 
> broker, the request will be handled as before: by only returning coordinator 
> host and port information.
> {code}
> GroupMetadataRequest => GroupId IncludeMetadata
>   GroupId => String
>   IncludeMetadata => Boolean
> GroupMetadataResponse => ErrorCode Coordinator GroupMetadata
>   ErrorCode => int16
>   Coordinator => Id Host Port
> Id => int32
> Host => string
> Port => int32
>   GroupMetadata => State ProtocolType Generation Protocol Leader  Members
> State => String
> ProtocolType => String
> Generation => int32
> Protocol => String
> Leader => String
> Members => [Member MemberMetadata MemberAssignment]
>   Member => MemberIp ClientId
> MemberIp => String
> ClientId => String
>   MemberMetadata => Bytes
>   MemberAssignment => Bytes
> {code}
> The request schema includes a flag to indicate whether metadata is needed, 
> which saves clients from having to read all group metadata when they are just 
> trying to find the coordinator. This is important to reduce group overhead 
> for use cases which involve a large number of topic subscriptions (e.g. 
> mirror maker).
> Tools will use the protocol type to determine how to parse metadata. For 
> example, when the protocolType is "consumer", the tool can use 
> ConsumerProtocol to parse the member metadata as topic subscriptions and 
> partition assignments. 
> The detailed proposal can be found below.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-40%3A+ListGroups+and+DescribeGroup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2738) Can't set SSL as inter-broker-protocol by rolling restart of brokers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991228#comment-14991228
 ] 

ASF GitHub Bot commented on KAFKA-2738:
---

GitHub user gwenshap opened a pull request:

https://github.com/apache/kafka/pull/428

KAFKA-2738: Replica FetcherThread should connect to leader endpoint m…

…atching its inter-broker security protocol

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2738

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/428.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #428


commit 994fa156a0f6e818184182e7e5c852ece58191ce
Author: Gwen Shapira 
Date:   2015-11-05T06:30:13Z

KAFKA-2738: Replica FetcherThread should connect to leader endpoint 
matching its inter-broker security protocol




> Can't set SSL as inter-broker-protocol by rolling restart of brokers
> 
>
> Key: KAFKA-2738
> URL: https://issues.apache.org/jira/browse/KAFKA-2738
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Scenario (as carefully documented by [~benstopford]:
> 1. Start 2 or more brokers with listeners on both PLAINTEXT and SSL 
> protocols, and PLAINTEXT as security.inter.broker.protocol:
> inter.broker.protocol.version = 0.9.0.X
> security.inter.broker.protocol = PLAINTEXT
> listeners = PLAINTEXT://:9092,SSL://:9093
> 2. Stop one of the brokers and change security.inter.broker.protocol to SSL
> inter.broker.protocol.version = 0.9.0.X
> security.inter.broker.protocol = SSL
> listeners = PLAINTEXT://:9092,SSL://:9093
> 3. Start that broker again.
> You will get replication errors as it will attempt to use SSL on a PLAINTEXT 
> port:
> {code}
> WARN ReplicaFetcherThread-0-3, Error in fetch 
> kafka.server.ReplicaFetcherThread$FetchRequest@78ca3ba1. Possible cause: 
> java.io.IOException: Connection to Node(3, worker4, 9092) failed 
> (kafka.server.ReplicaFetcherThread)
> WARN Failed to send SSL Close message 
> (org.apache.kafka.common.network.SslTransportLayer)
> java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:492)
> at 
> org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:188)
> at 
> org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:161)
> at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:50)
> at org.apache.kafka.common.network.Selector.close(Selector.java:448)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:316)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:128)
> at 
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
> at 
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:105)
> at 
> kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:58)
> at 
> kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:202)
> at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:192)
> at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:102)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:93)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2744) WorkerSourceTask commits offsets too early when stopping

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990286#comment-14990286
 ] 

ASF GitHub Bot commented on KAFKA-2744:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/423


> WorkerSourceTask commits offsets too early when stopping
> 
>
> Key: KAFKA-2744
> URL: https://issues.apache.org/jira/browse/KAFKA-2744
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> The call to commit offsets appears ok at first glance because we've invoked 
> the SourceTask's stop method, but it runs before we've stopped the work 
> thread which could still be invoking the SourceTask's poll() method and may 
> have outstanding data. We need to wait until we're sure the work thread has 
> completely finished so we're guaranteed to flush all the data generated by 
> the SourceTask and commit the final offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2691) Improve handling of authorization failure during metadata refresh

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990190#comment-14990190
 ] 

ASF GitHub Bot commented on KAFKA-2691:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/394


> Improve handling of authorization failure during metadata refresh
> -
>
> Key: KAFKA-2691
> URL: https://issues.apache.org/jira/browse/KAFKA-2691
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> There are two problems, one more severe than the other:
> 1. The consumer blocks indefinitely if there is non-transient authorization 
> failure during metadata refresh due to KAFKA-2391
> 2. We get a TimeoutException instead of an AuthorizationException in the 
> producer for the same case
> If the fix for KAFKA-2391 is to add a timeout, then we will have issue `2` in 
> both producer and consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2741) Source/SinkTaskContext should be interfaces and implementations kept in runtime jar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990235#comment-14990235
 ] 

ASF GitHub Bot commented on KAFKA-2741:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/420

KAFKA-2741: Make SourceTaskContext and SinkTaskContext interfaces and keep 
implementations in runtime jar.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka task-context-interfaces

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #420


commit bf024989a242f5132b7418a8c163ae4e922d06bd
Author: Ewen Cheslack-Postava 
Date:   2015-11-04T18:53:02Z

KAFKA-2741: Make SourceTaskContext and SinkTaskContext interfaces and keep 
implementations in runtime jar.




> Source/SinkTaskContext should be interfaces and implementations kept in 
> runtime jar
> ---
>
> Key: KAFKA-2741
> URL: https://issues.apache.org/jira/browse/KAFKA-2741
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> Over time, these got turned into abstract classes, but they should just be 
> interfaces with the specific implementation kept entirely in the runtime jar 
> and opaque to connectors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2743) Forwarding task reconfigurations in Copycat can deadlock with rebalances and has no backoff

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990238#comment-14990238
 ] 

ASF GitHub Bot commented on KAFKA-2743:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/422

KAFKA-2743: Make forwarded task reconfiguration requests asynchronous, run 
on a separate thread, and backoff before retrying when they fail.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
task-reconfiguration-async-with-backoff

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #422


commit 8a30a78b9222ed8fec5143a41db5cf8e6e9efbc7
Author: Ewen Cheslack-Postava 
Date:   2015-11-03T05:30:32Z

KAFKA-2743: Make forwarded task reconfiguration requests asynchronous, run 
on a separate thread, and backoff before retrying when they fail.




> Forwarding task reconfigurations in Copycat can deadlock with rebalances and 
> has no backoff
> ---
>
> Key: KAFKA-2743
> URL: https://issues.apache.org/jira/browse/KAFKA-2743
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> There are two issues with the way we're currently forwarding task 
> reconfigurations. First, the forwarding is performed synchronously in the 
> DistributedHerder's main processing loop. If node A forwards a task 
> reconfiguration and node B has started a rebalance process, we can end up 
> with distributed deadlock because node A will be blocking on the HTTP request 
> in the thread that would otherwise handle heartbeating and rebalancing.
> Second, currently we just retry aggressively with no backoff. In some cases 
> the node that is currently thought to be the leader will legitimately be down 
> (it shutdown and the node sending the request didn't rebalance yet), so we 
> need some backoff to avoid unnecessarily hammering the network and the huge 
> log files that result from constant errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2744) WorkerSourceTask commits offsets too early when stopping

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990239#comment-14990239
 ] 

ASF GitHub Bot commented on KAFKA-2744:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/423

KAFKA-2744: Commit source task offsets after task is completely stopped to 
ensure no additional messages are processed during the offset commit when 
stopping tasks for rebalancing.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
commit-source-offsets-after-work-thread-exits

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #423


commit 5c44efa57215153d8d8b3ca6ad50a5f878602f79
Author: Ewen Cheslack-Postava 
Date:   2015-11-04T05:42:23Z

KAFKA-2744: Commit source task offsets after task is completely stopped to 
ensure no additional messages are processed during the offset commit when 
stopping tasks for rebalancing.




> WorkerSourceTask commits offsets too early when stopping
> 
>
> Key: KAFKA-2744
> URL: https://issues.apache.org/jira/browse/KAFKA-2744
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> The call to commit offsets appears ok at first glance because we've invoked 
> the SourceTask's stop method, but it runs before we've stopped the work 
> thread which could still be invoking the SourceTask's poll() method and may 
> have outstanding data. We need to wait until we're sure the work thread has 
> completely finished so we're guaranteed to flush all the data generated by 
> the SourceTask and commit the final offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2742) SourceTaskOffsetCommitter does not properly remove commit tasks when they are already in progress

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990236#comment-14990236
 ] 

ASF GitHub Bot commented on KAFKA-2742:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/421

KAFKA-2742: Fix SourceTaskOffsetCommitter to handle removal of commit tasks 
when they are already in progress.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
wait-on-in-progress-source-offset-commits

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/421.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #421


commit 023b3be05978ee714955fdfc04809b7e286aaca1
Author: Ewen Cheslack-Postava 
Date:   2015-11-03T05:26:39Z

KAFKA-2742: Fix SourceTaskOffsetCommitter to handle removal of commit tasks 
when they are already in progress.




> SourceTaskOffsetCommitter does not properly remove commit tasks when they are 
> already in progress
> -
>
> Key: KAFKA-2742
> URL: https://issues.apache.org/jira/browse/KAFKA-2742
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> The current implementation is relying on ScheduledExecutorService to cancel 
> the task, but this doesn't handle in-progress tasks and can result in 
> stopping source tasks not completing a final offset commit before considering 
> the task fully stopped. This can allow rebalancing to proceed before offsets 
> are fully committed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2742) SourceTaskOffsetCommitter does not properly remove commit tasks when they are already in progress

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991938#comment-14991938
 ] 

ASF GitHub Bot commented on KAFKA-2742:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/421


> SourceTaskOffsetCommitter does not properly remove commit tasks when they are 
> already in progress
> -
>
> Key: KAFKA-2742
> URL: https://issues.apache.org/jira/browse/KAFKA-2742
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> The current implementation is relying on ScheduledExecutorService to cancel 
> the task, but this doesn't handle in-progress tasks and can result in 
> stopping source tasks not completing a final offset commit before considering 
> the task fully stopped. This can allow rebalancing to proceed before offsets 
> are fully committed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2738) Can't set SSL as inter-broker-protocol by rolling restart of brokers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992000#comment-14992000
 ] 

ASF GitHub Bot commented on KAFKA-2738:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/428


> Can't set SSL as inter-broker-protocol by rolling restart of brokers
> 
>
> Key: KAFKA-2738
> URL: https://issues.apache.org/jira/browse/KAFKA-2738
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Scenario (as carefully documented by [~benstopford]:
> 1. Start 2 or more brokers with listeners on both PLAINTEXT and SSL 
> protocols, and PLAINTEXT as security.inter.broker.protocol:
> inter.broker.protocol.version = 0.9.0.X
> security.inter.broker.protocol = PLAINTEXT
> listeners = PLAINTEXT://:9092,SSL://:9093
> 2. Stop one of the brokers and change security.inter.broker.protocol to SSL
> inter.broker.protocol.version = 0.9.0.X
> security.inter.broker.protocol = SSL
> listeners = PLAINTEXT://:9092,SSL://:9093
> 3. Start that broker again.
> You will get replication errors as it will attempt to use SSL on a PLAINTEXT 
> port:
> {code}
> WARN ReplicaFetcherThread-0-3, Error in fetch 
> kafka.server.ReplicaFetcherThread$FetchRequest@78ca3ba1. Possible cause: 
> java.io.IOException: Connection to Node(3, worker4, 9092) failed 
> (kafka.server.ReplicaFetcherThread)
> WARN Failed to send SSL Close message 
> (org.apache.kafka.common.network.SslTransportLayer)
> java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:492)
> at 
> org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:188)
> at 
> org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:161)
> at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:50)
> at org.apache.kafka.common.network.Selector.close(Selector.java:448)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:316)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:128)
> at 
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
> at 
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:105)
> at 
> kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:58)
> at 
> kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:202)
> at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:192)
> at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
> at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:102)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:93)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2741) Source/SinkTaskContext should be interfaces and implementations kept in runtime jar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991943#comment-14991943
 ] 

ASF GitHub Bot commented on KAFKA-2741:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/420


> Source/SinkTaskContext should be interfaces and implementations kept in 
> runtime jar
> ---
>
> Key: KAFKA-2741
> URL: https://issues.apache.org/jira/browse/KAFKA-2741
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> Over time, these got turned into abstract classes, but they should just be 
> interfaces with the specific implementation kept entirely in the runtime jar 
> and opaque to connectors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2743) Forwarding task reconfigurations in Copycat can deadlock with rebalances and has no backoff

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991949#comment-14991949
 ] 

ASF GitHub Bot commented on KAFKA-2743:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/422


> Forwarding task reconfigurations in Copycat can deadlock with rebalances and 
> has no backoff
> ---
>
> Key: KAFKA-2743
> URL: https://issues.apache.org/jira/browse/KAFKA-2743
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> There are two issues with the way we're currently forwarding task 
> reconfigurations. First, the forwarding is performed synchronously in the 
> DistributedHerder's main processing loop. If node A forwards a task 
> reconfiguration and node B has started a rebalance process, we can end up 
> with distributed deadlock because node A will be blocking on the HTTP request 
> in the thread that would otherwise handle heartbeating and rebalancing.
> Second, currently we just retry aggressively with no backoff. In some cases 
> the node that is currently thought to be the leader will legitimately be down 
> (it shutdown and the node sending the request didn't rebalance yet), so we 
> need some backoff to avoid unnecessarily hammering the network and the huge 
> log files that result from constant errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2702) ConfigDef toHtmlTable() sorts in a way that is a bit confusing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992063#comment-14992063
 ] 

ASF GitHub Bot commented on KAFKA-2702:
---

Github user granthenke closed the pull request at:

https://github.com/apache/kafka/pull/379


> ConfigDef toHtmlTable() sorts in a way that is a bit confusing
> --
>
> Key: KAFKA-2702
> URL: https://issues.apache.org/jira/browse/KAFKA-2702
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Grant Henke
> Attachments: ConsumerConfig-After-v2.html, ConsumerConfig-After.html, 
> ConsumerConfig-Before.html
>
>
> Because we put everything without default first (without prioritizing), 
> critical  parameters get placed below low priority ones when they both have 
> no defaults. Some parameters are without default and optional (SASL server in 
> ConsumerConfig for instance).
> Try printing ConsumerConfig parameters and see the mandatory group.id show up 
> as #15.
> I suggest sorting the no-default parameters by priority as well, or perhaps 
> adding a "REQUIRED" category that gets printed first no matter what.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2702) ConfigDef toHtmlTable() sorts in a way that is a bit confusing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992064#comment-14992064
 ] 

ASF GitHub Bot commented on KAFKA-2702:
---

GitHub user granthenke reopened a pull request:

https://github.com/apache/kafka/pull/379

KAFKA-2702: ConfigDef toHtmlTable() sorts in a way that is a bit conf…

…using

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka config-html

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/379.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #379


commit bc17f6b923f63891c1c36c531b525783df834e42
Author: Grant Henke 
Date:   2015-10-29T03:49:24Z

KAFKA-2702: ConfigDef toHtmlTable() sorts in a way that is a bit confusing




> ConfigDef toHtmlTable() sorts in a way that is a bit confusing
> --
>
> Key: KAFKA-2702
> URL: https://issues.apache.org/jira/browse/KAFKA-2702
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Grant Henke
> Attachments: ConsumerConfig-After-v2.html, ConsumerConfig-After.html, 
> ConsumerConfig-Before.html
>
>
> Because we put everything without default first (without prioritizing), 
> critical  parameters get placed below low priority ones when they both have 
> no defaults. Some parameters are without default and optional (SASL server in 
> ConsumerConfig for instance).
> Try printing ConsumerConfig parameters and see the mandatory group.id show up 
> as #15.
> I suggest sorting the no-default parameters by priority as well, or perhaps 
> adding a "REQUIRED" category that gets printed first no matter what.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2748) SinkTasks do not handle rebalances and offset commit properly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992060#comment-14992060
 ] 

ASF GitHub Bot commented on KAFKA-2748:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/431

KAFKA-2748: Ensure sink tasks commit offsets upon rebalance and rewind if 
the SinkTask flush fails.

Also fix the incorrect consumer group ID setting which was giving each task 
its
own group instead of one for the entire sink connector.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2748-sink-task-rebalance-commit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/431.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #431


commit 4fc35c2fd6f358b9d234bf9338a7ef342c87af83
Author: Ewen Cheslack-Postava 
Date:   2015-11-05T01:04:56Z

KAFKA-2748: Ensure sink tasks commit offsets upon rebalance and rewind if 
the SinkTask flush fails.

Also fix the incorrect consumer group ID setting which was giving each task 
its
own group instead of one for the entire sink connector.




> SinkTasks do not handle rebalances and offset commit properly
> -
>
> Key: KAFKA-2748
> URL: https://issues.apache.org/jira/browse/KAFKA-2748
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> Since the initial SinkTask code was originally written with an early version 
> of the new consumer, it wasn't setup to handle rebalances properly. Since we 
> recently added the rebalance listener, we can use it to correctly commit 
> offsets. However, the existing code also has two issues. First, in the case 
> of a failure to flush data in the sink task, we are not correctly rewinding 
> to the last committed offsets. We need to do this since we cannot be sure 
> what happened to the outstanding data, so we need to reprocess it. 
> Second, flushing when stopping was not being handled propertly. The existing 
> code was assuming that as part of SinkTask.stop() we would. However, this did 
> not make sense since SinkTask.stop() was being invoked before the worker 
> thread was stopped, so we could end up committing the wrong offsets. Instead, 
> we need to wait for the worker thread to finish whatever it is currently 
> doing, do one final flush + commit offsets, and only then invoke stop() to 
> allow the task to do final cleanup. This is a bit confusing because stop 
> means different things for source and sink tasks since they have pull vs push 
> semantics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2572) zk connection instability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992082#comment-14992082
 ] 

ASF GitHub Bot commented on KAFKA-2572:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/432

KAFKA-2572: Add VerifiableSource/Sink connectors and rolling bounce Copycat 
system tests.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2752-copycat-clean-bounce-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/432.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #432


commit 1adc1f8a405bd1248214863a34c5475397bf1c5e
Author: Ewen Cheslack-Postava 
Date:   2015-11-03T05:25:17Z

KAFKA-2572: Add VerifiableSource/Sink connectors and rolling bounce Copycat 
system tests.




> zk connection instability
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log, 
> 091115-full.log.zip
>
>
> On several occasions, we've seen our process enter a cycle of: zk session 
> expiry; new session creation; rebalancing activity; pause during which 
> nothing is heard from the zk server. Sometimes, the reconnections are 
> successful, elements are pulled from Kafka, but then disconnection and 
> reconnection occurs shortly thereafter, causing OOMs when new elements are 
> pulled in (although OOMs were not seen in the two cases attached as 
> examples). Restarting the process that uses the zk client resolved the 
> problems in both cases.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show 
> all logs entries minus entries particular to our application. For 09/08, the 
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; 
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining 
> only error and warning entries, and entries containing any of: "begin 
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the 
> 09/11 digest logs, entries from the kafka.network.Processor logger are also 
> excised for clarity. Unfortunately, debug logging was not enabled during 
> these events.
> The 09/11 case shows repeated cycles of session expiry, followed by 
> rebalancing activity, followed by a pause during which nothing is heard from 
> the zk server, followed by a session timeout. A stable session seems to have 
> been established at 2015-09-11T04:13:47.140-04:00, but messages of the form 
> "I wrote this conflicted ephemeral node 
> [{"version":1,"subscription":{"binlogs_mailchimp_us2":100},"pattern":"static","timestamp":"1441959227564"}]
>  at 
> /consumers/prologue-second-stage_prod_us2/ids/prologue-second-stage_prod_us2_app01.c1.prologue.prod.atl01.rsglab.com-1441812334972-b967b718
>  a while back in a different session, hence I will backoff for this node to 
> be deleted by Zookeeper and retry" were logged out repeatedly until we 
> restarted the process after 2015-09-11T07:44:47.124-04:00, which marks the 
> final entry in the log.
> The 09/08 case is a little more straightforward than the 09/11 case, in that 
> a stable session was not established prior to our restarting the process.
> It's perhaps also noteworthy that in the 09/08 case, two timeouts for the 
> same session are seen during a single rebalance, at 
> 2015-09-08T12:52:19.107-04:00 and 2015-09-08T12:52:31.639-04:00. The 
> rebalance in question begins at 2015-09-08T12:52:06.667-04:00.
> The connection to ZK expires and is restablished multiple times before the 
> process is killed after 2015-09-08T13:13:41.655-04:00, which marks the last 
> entry in the logs for this day.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2768) New-consumer sends invalid describeGroupResponse while restabilizing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994599#comment-14994599
 ] 

ASF GitHub Bot commented on KAFKA-2768:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/447

KAFKA-2768: New-consumer sends invalid describeGroupResponse while re…

…stabilizing

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2768

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/447.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #447


commit 547f1b2f1c46cc4c09bbd334bcc9e1db7fc7cbd8
Author: Ashish Singh 
Date:   2015-11-06T22:34:38Z

KAFKA-2768: New-consumer sends invalid describeGroupResponse while 
restabilizing




> New-consumer sends invalid describeGroupResponse while restabilizing
> 
>
> Key: KAFKA-2768
> URL: https://issues.apache.org/jira/browse/KAFKA-2768
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> While a consumer group is restabilizing, it sends describeGroupResponse with 
> no error, but empty metadata and assignments. That leads to SchemaException 
> on client.
> {code}
> Error while executing consumer group command Error reading field 'version': 
> java.nio.BufferUnderflowException
> org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
> 'version': java.nio.BufferUnderflowException
>   at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerProtocol.deserializeAssignment(ConsumerProtocol.java:106)
>   at 
> kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:163)
>   at 
> kafka.admin.AdminClient$$anonfun$describeConsumerGroup$1.apply(AdminClient.scala:161)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:161)
>   at 
> kafka.admin.ConsumerGroupCommand$.describe(ConsumerGroupCommand.scala:109)
>   at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:63)
>   at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2765) Connectors and tasks should have versions that can be reported when they are instantiated

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994863#comment-14994863
 ] 

ASF GitHub Bot commented on KAFKA-2765:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/446


> Connectors and tasks should have versions that can be reported when they are 
> instantiated
> -
>
> Key: KAFKA-2765
> URL: https://issues.apache.org/jira/browse/KAFKA-2765
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> Including version information will allow for a few things:
> * Basic logging of versions can help with debugging/configuration issues if 
> you have old jars on your classpath.
> * Can allow for Copycat to aid in the version upgrade process when you want 
> to update to a newer version of a connector.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2766) Make the Java Producer as the default producer in tooling.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994871#comment-14994871
 ] 

ASF GitHub Bot commented on KAFKA-2766:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/448


> Make the Java Producer as the default producer in tooling.
> --
>
> Key: KAFKA-2766
> URL: https://issues.apache.org/jira/browse/KAFKA-2766
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.9.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)