[jira] [Created] (KAFKA-5993) Kafka AdminClient does not support standard security settings

[Stephane Maarek (JIRA)]

Stephane Maarek created KAFKA-5993:
--

 Summary: Kafka AdminClient does not support standard security 
settings
 Key: KAFKA-5993
 URL: https://issues.apache.org/jira/browse/KAFKA-5993
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.1
Reporter: Stephane Maarek


Kafka Admin Client does not support basic security configurations, such as 
"sasl.jaas.config".
Therefore it makes it impossible to use against a secure cluster

```
14:12:12.948 [main] WARN  org.apache.kafka.clients.admin.AdminClientConfig - 
The configuration 'sasl.jaas.config' was supplied but isn't a known config.
```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5992) Better Java Documentation for AdminClient Exceptions

[Stephane Maarek (JIRA)]

Stephane Maarek created KAFKA-5992:
--

 Summary: Better Java Documentation for AdminClient Exceptions
 Key: KAFKA-5992
 URL: https://issues.apache.org/jira/browse/KAFKA-5992
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.1
Reporter: Stephane Maarek


When invoking a describeTopics operation on a topic that does not exist, we get 
an InvalidTopicException as a RuntimeException.

I believe this should be documented, and the API maybe changed:

For example changing:
{code:java}
public DescribeTopicsResult describeTopics(Collection topicNames) {
{code}

To:
{code:java}
public DescribeTopicsResult describeTopics(Collection topicNames) 
throws InvalidTopicException 
{code}

Additionally, in case multiple topics don't exist, only the first one will 
throw an error. This is really not scalable. 

Maybe the DescribeTopicsResult could have a Boolean "topicExists" ? 
Up for discussion





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5565) Add a broker metric specifying the number of consumer group rebalances in progress

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5565:
---
Fix Version/s: 1.0.0

> Add a broker metric specifying the number of consumer group rebalances in 
> progress
> --
>
> Key: KAFKA-5565
> URL: https://issues.apache.org/jira/browse/KAFKA-5565
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 1.0.0
>
>
> We should add a broker metric specifying the number of consumer group 
> rebalances in progress.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-4972) Kafka 0.10.0 Found a corrupted index file during Kafka broker startup

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4972:
---
Fix Version/s: 1.0.0

> Kafka 0.10.0  Found a corrupted index file during Kafka broker startup
> --
>
> Key: KAFKA-4972
> URL: https://issues.apache.org/jira/browse/KAFKA-4972
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
> Environment: JDK: HotSpot  x64  1.7.0_80
> Tag: 0.10.0
>Reporter: fangjinuo
>Priority: Critical
> Fix For: 1.0.0, 0.11.0.2
>
> Attachments: Snap3.png
>
>
> -deleted text-After force shutdown all kafka brokers one by one, restart them 
> one by one, but a broker startup failure.
> The following WARN leval log was found in the log file:
> found a corrutped index file,  .index , delet it  ...
> you can view details by following attachment.
> ~I look up some codes in core module, found out :
> the nonthreadsafe method LogSegment.append(offset, messages)  has tow caller:
> 1) Log.append(messages)  // here has a synchronized 
> lock 
> 2) LogCleaner.cleanInto(topicAndPartition, source, dest, map, retainDeletes, 
> messageFormatVersion)   // here has not 
> So I guess this may be the reason for the repeated offset in 0xx.log file 
> (logsegment's .log) ~
> Although this is just my inference, but I hope that this problem can be 
> quickly repaired



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5957) Producer IllegalStateException due to second deallocate after aborting a batch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185267#comment-16185267
 ] 

ASF GitHub Bot commented on KAFKA-5957:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3942


> Producer IllegalStateException due to second deallocate after aborting a batch
> --
>
> Key: KAFKA-5957
> URL: https://issues.apache.org/jira/browse/KAFKA-5957
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 1.0.0
>
>
> Saw this recently in a system test failure:
> {code}
> [2017-09-21 05:04:52,033] ERROR [Producer clientId=producer-1, 
> transactionalId=my-second-transactional-id] Aborting producer batches due to 
> fatal error (org.apache.kafka.clients.producer.internals.Sender)
> org.apache.kafka.common.KafkaException: The client hasn't received 
> acknowledgment for some previously sent messages and can no longer retry 
> them. It isn't safe to continue.
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:211)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:164)
> at java.lang.Thread.run(Thread.java:745)
> [2017-09-21 05:04:52,033] TRACE Aborting batch for partition output-topic-2 
> (org.apache.kafka.clients.producer.internals.ProducerBatch)
> org.apache.kafka.common.KafkaException: The client hasn't received 
> acknowledgment for some previously sent messages and can no longer retry 
> them. It isn't safe to continue.
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:211)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:164)
> at java.lang.Thread.run(Thread.java:745)
> [2017-09-21 05:04:52,134] TRACE [Producer clientId=producer-1, 
> transactionalId=my-second-transactional-id] Not sending transactional request 
> (type=EndTxnRequest, transactionalId=my-second-transactional-id, 
> producerId=1000, producerEpoch=0, result=COMMIT) because we are in an error 
> state (org.apache.kafka.clients.producer.internals.TransactionManager)
> [2017-09-21 05:04:52,134] INFO [Producer clientId=producer-1, 
> transactionalId=my-second-transactional-id] Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms. 
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2017-09-21 05:04:52,134] DEBUG [Producer clientId=producer-1, 
> transactionalId=my-second-transactional-id] Beginning shutdown of Kafka 
> producer I/O thread, sending remaining records. 
> (org.apache.kafka.clients.producer.internals.Sender)
> [2017-09-21 05:04:52,360] TRACE [Producer clientId=producer-1, 
> transactionalId=my-second-transactional-id] Received produce response from 
> node 1 with correlation id 245 
> (org.apache.kafka.clients.producer.internals.Sender)
> [2017-09-21 05:04:52,360] DEBUG [Producer clientId=producer-1, 
> transactionalId=my-second-transactional-id] ProducerId: 1000; Set last ack'd 
> sequence number for topic-partition output-topic-2 to 136 
> (org.apache.kafka.clients.producer.internals.Sender)
> [2017-09-21 05:04:52,360] TRACE Successfully produced messages to 
> output-topic-2 with base offset 387. 
> (org.apache.kafka.clients.producer.internals.ProducerBatch)
> [2017-09-21 05:04:52,360] DEBUG ProduceResponse returned for output-topic-2 
> after batch had already been aborted. 
> (org.apache.kafka.clients.producer.internals.ProducerBatch)
> [2017-09-21 05:04:52,360] ERROR [Producer clientId=producer-1, 
> transactionalId=my-second-transactional-id] Uncaught error in request 
> completion: (org.apache.kafka.clients.NetworkClient)
> java.lang.IllegalStateException: Remove from the incomplete set failed. This 
> should be impossible.
> at 
> org.apache.kafka.clients.producer.internals.IncompleteBatches.remove(IncompleteBatches.java:44)
> at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.deallocate(RecordAccumulator.java:612)
> at 
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:585)
> at 
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:561)
> at 
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:475)
> at 
> org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:74)
> at 
> org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:685)
> at 
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:101)
> at 
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:481)
> at 

[jira] [Commented] (KAFKA-5746) Add new metrics to support health checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185235#comment-16185235
 ] 

ASF GitHub Bot commented on KAFKA-5746:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3989

KAFKA-5746; Fix conversion count computed in `downConvert`

It should be the number of records instead of the
number of batches.

A few additional clean-ups: minor renames,
removal of unused code, test fixes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-5746-health-metrics-follow-up

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3989.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3989


commit a85feb150a4aea8d879535c27ddb857278790e63
Author: Ismael Juma 
Date:   2017-09-29T01:40:29Z

KAFKA-5746; Fix downConvert's conversion count

It should be the number of records instead of the
number of batches.

A few additional clean-ups: minor renames,
removal of unused code, test fixes.




> Add new metrics to support health checks
> 
>
> Key: KAFKA-5746
> URL: https://issues.apache.org/jira/browse/KAFKA-5746
> Project: Kafka
>  Issue Type: New Feature
>  Components: metrics
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> It will be useful to have some additional metrics to support health checks.
> Details are in 
> [KIP-188|https://cwiki.apache.org/confluence/display/KAFKA/KIP-188+-+Add+new+metrics+to+support+health+checks]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5989) disableLogging() causes an initialization loop

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-5989.

Resolution: Duplicate

\cc [~damianguy] It this is not a duplicate, please reopen.

> disableLogging() causes an initialization loop
> --
>
> Key: KAFKA-5989
> URL: https://issues.apache.org/jira/browse/KAFKA-5989
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.1
>Reporter: Tuan Nguyen
> Attachments: App.java
>
>
> Using {{disableLogging()}} for either of the built-in state store types 
> causes an initialization loop in the StreamThread.
> Case A - this works just fine:
> {code}
>   final StateStoreSupplier testStore = Stores.create(topic)
>   .withStringKeys()
>   .withStringValues()
>   .inMemory()
> //.disableLogging() 
>   .maxEntries(10)
>   .build();
> {code}
> Case B - this does not:
> {code}
>   final StateStoreSupplier testStore = Stores.create(topic)
>   .withStringKeys()
>   .withStringValues()
>   .inMemory()
>   .disableLogging() 
>   .maxEntries(10)
>   .build();
> {code}
> A brief debugging dive shows that in Case B, 
> {{AssignedTasks.allTasksRunning()}} never returns true, because of a remnant 
> entry in {{AssignedTasks#restoring}} that never gets properly restored.
> See [^App.java] for a working test (requires ZK + Kafka ensemble, and at 
> least one keyed message produced to the "test" topic)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185167#comment-16185167
 ] 

Matthias J. Sax commented on KAFKA-5967:


It's correct. But I did not see any email from you. You need to subscribe to 
the list first I think: http://kafka.apache.org/contact

> Ineffective check of negative value in 
> CompositeReadOnlyKeyValueStore#approximateNumEntries()
> -
>
> Key: KAFKA-5967
> URL: https://issues.apache.org/jira/browse/KAFKA-5967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>  Labels: beginner, newbie
>
> {code}
> long total = 0;
> for (ReadOnlyKeyValueStore store : stores) {
> total += store.approximateNumEntries();
> }
> return total < 0 ? Long.MAX_VALUE : total;
> {code}
> The check for negative value seems to account for wrapping.
> However, wrapping can happen within the for loop. So the check should be 
> performed inside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5473) handle ZK session expiration properly when a new session can't be established

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185158#comment-16185158
 ] 

Ismael Juma commented on KAFKA-5473:


[~prasincs], can you update this JIRA tomorrow with your progress then? If you 
can get something by Saturday, I think that would work. Thanks!

> handle ZK session expiration properly when a new session can't be established
> -
>
> Key: KAFKA-5473
> URL: https://issues.apache.org/jira/browse/KAFKA-5473
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Prasanna Gautam
> Fix For: 1.0.0
>
>
> In https://issues.apache.org/jira/browse/KAFKA-2405, we change the logic in 
> handling ZK session expiration a bit. If a new ZK session can't be 
> established after session expiration, we just log an error and continue. 
> However, this can leave the broker in a bad state since it's up, but not 
> registered from the controller's perspective. Replicas on this broker may 
> never to be in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5473) handle ZK session expiration properly when a new session can't be established

[Prasanna Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185144#comment-16185144
 ] 

Prasanna Gautam commented on KAFKA-5473:


Yeah, I'm OK if someone can pick it up too. I'm aiming for sometime around 
tomorrow or over the weekend for a PR. Will be happy to review, test and help 
in any way I can.

> handle ZK session expiration properly when a new session can't be established
> -
>
> Key: KAFKA-5473
> URL: https://issues.apache.org/jira/browse/KAFKA-5473
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Prasanna Gautam
> Fix For: 1.0.0
>
>
> In https://issues.apache.org/jira/browse/KAFKA-2405, we change the logic in 
> handling ZK session expiration a bit. If a new ZK session can't be 
> established after session expiration, we just log an error and continue. 
> However, this can leave the broker in a bad state since it's up, but not 
> registered from the controller's perspective. Replicas on this broker may 
> never to be in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()

[siva santhalingam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185128#comment-16185128
 ] 

siva santhalingam commented on KAFKA-5967:
--

[~mjsax] I sent an email to d...@kafka.apache.org. Please let me know if thats 
the right email alias. Thanks!

> Ineffective check of negative value in 
> CompositeReadOnlyKeyValueStore#approximateNumEntries()
> -
>
> Key: KAFKA-5967
> URL: https://issues.apache.org/jira/browse/KAFKA-5967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>  Labels: beginner, newbie
>
> {code}
> long total = 0;
> for (ReadOnlyKeyValueStore store : stores) {
> total += store.approximateNumEntries();
> }
> return total < 0 ? Long.MAX_VALUE : total;
> {code}
> The check for negative value seems to account for wrapping.
> However, wrapping can happen within the for loop. So the check should be 
> performed inside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185119#comment-16185119
 ] 

ASF GitHub Bot commented on KAFKA-5967:
---

GitHub user shivsantham opened a pull request:

https://github.com/apache/kafka/pull/3988

KAFKA-5967 Ineffective check of negative value in CompositeReadOnlyKe…

package name: org.apache.kafka.streams.state.internals
Minor change to approximateNumEntries() method in 
CompositeReadOnlyKeyValueStore class.

long total = 0;
   for (ReadOnlyKeyValueStore store : stores) {
  total += store.approximateNumEntries();
   }

return total < 0 ? Long.MAX_VALUE : total;

The check for negative value seems to account for wrapping. However, 
wrapping can happen within the for loop. So the check should be performed 
inside the loop.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shivsantham/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3988.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3988


commit ff377759a943c7bfb89a56ad721e7ba1b3b0b24c
Author: siva santhalingam 
Date:   2017-09-28T23:37:47Z

KAFKA-5967 Ineffective check of negative value in 
CompositeReadOnlyKeyValueStore#approximateNumEntries()

long total = 0;
   for (ReadOnlyKeyValueStore store : stores) {
  total += store.approximateNumEntries();
   }

return total < 0 ? Long.MAX_VALUE : total;

The check for negative value seems to account for wrapping. However, 
wrapping can happen within the for loop. So the check should be performed 
inside the loop.




> Ineffective check of negative value in 
> CompositeReadOnlyKeyValueStore#approximateNumEntries()
> -
>
> Key: KAFKA-5967
> URL: https://issues.apache.org/jira/browse/KAFKA-5967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>  Labels: beginner, newbie
>
> {code}
> long total = 0;
> for (ReadOnlyKeyValueStore store : stores) {
> total += store.approximateNumEntries();
> }
> return total < 0 ? Long.MAX_VALUE : total;
> {code}
> The check for negative value seems to account for wrapping.
> However, wrapping can happen within the for loop. So the check should be 
> performed inside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5990) Add generated documentation for Connect metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185118#comment-16185118
 ] 

ASF GitHub Bot commented on KAFKA-5990:
---

GitHub user rhauch opened a pull request:

https://github.com/apache/kafka/pull/3987

KAFKA-5990: Enable generation of metrics docs for Connect

A new mechanism was added recently to the Metrics framework to make it 
easier to generate the documentation. It uses a registry with a 
MetricsNameTemplate for each metric, and then those templates are used when 
creating the actual metrics. The metrics framework provides utilities that can 
generate the HTML documentation from the registry of templates.

This change moves the recently-added Connect metrics over to use these 
templates and to then generate the metric documentation for Connect.

This PR is based upon #3975 and can be rebased once that has been merged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rhauch/kafka kafka-5990

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3987.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3987


commit 23aebcdf7797923865a771e7acac89e7e2572e3d
Author: Randall Hauch 
Date:   2017-09-26T15:28:12Z

KAFKA-5902 Added sink task metrics

commit 9824e329ef599c59ea4e7f60cf46ec907b516d90
Author: Randall Hauch 
Date:   2017-09-28T20:47:39Z

KAFKA-5902 Changed to measuring task processing lag behind consumer

Changed the `sink-record-lag-max` metric to be the maximum lag in terms of 
number of records that the sink task is behind the consumer's position for any 
topic partitions. This is not ideal, since often “lag” is defined to represent 
how far behind the task (or consumer) is relative to the end of the topic 
partition. However, the most recent offset for the topic partition is not easy 
to access in Connect.

commit d38dbde9f72926c68383729f9c80513879913cde
Author: Randall Hauch 
Date:   2017-09-28T19:12:21Z

KAFKA-5990 Enable generation of metrics docs for Connect

A new mechanism was added recently to the Metrics framework to make it 
easier to generate the documentation. It uses a registry with a 
MetricsNameTemplate for each metric, and then those templates are used when 
creating the actual metrics. The metrics framework provides utilities that can 
generate the HTML documentation from the registry of templates.

This change moves the recently-added Connect metrics over to use these 
templates and to then generate the metric documentation for Connect.




> Add generated documentation for Connect metrics
> ---
>
> Key: KAFKA-5990
> URL: https://issues.apache.org/jira/browse/KAFKA-5990
> Project: Kafka
>  Issue Type: Sub-task
>  Components: documentation, KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
> Fix For: 1.0.0
>
>
> KAFKA-5191 recently added a new {{MetricNameTemplate}} that is used to create 
> the {{MetricName}} objects in the producer and consumer, as we as in the 
> newly-added generation of metric documentation. The {{Metric.toHtmlTable}} 
> method then takes these templates and generates an HTML documentation for the 
> metrics.
> Change the Connect metrics to use these templates and update the build to 
> generate these metrics and include them in the Kafka documentation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-2359) New consumer - partitions auto assigned only on poll

[Nick West (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185102#comment-16185102
 ] 

Nick West commented on KAFKA-2359:
--

Work around, unsafe and brittle, but it works.

{code}
// Use reflection to get at the inner workings to ensure partitions have been 
created
Field subscriptionsField = 
consumer.getClass().getDeclaredField("subscriptions");
subscriptionsField.setAccessible(true);
Field coordinatorField = consumer.getClass().getDeclaredField("coordinator"); 
coordinatorField.setAccessible(true);

SubscriptionState subscriptions = 
(SubscriptionState)subscriptionsField.get(consumer);
ConsumerCoordinator coordinator = 
(ConsumerCoordinator)coordinatorField.get(consumer);

if (subscriptions.partitionsAutoAssigned())
coordinator.ensurePartitionAssignment();
{code}

> New consumer - partitions auto assigned only on poll
> 
>
> Key: KAFKA-2359
> URL: https://issues.apache.org/jira/browse/KAFKA-2359
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Stevo Slavic
>Priority: Minor
>
> In the new consumer I encountered unexpected behavior. After constructing 
> {{KafkaConsumer}} instance with configured consumer rebalance callback 
> handler, and subscribing to a topic with "consumer.subscribe(topic)", 
> retrieving subscriptions would return empty set and callback handler would 
> not get called (no partitions ever assigned or revoked), no matter how long 
> instance was up.
> Then I found by inspecting {{KafkaConsumer}} code that partition assignment 
> will only be triggered on first {{poll}}, since {{pollOnce}} has:
> {noformat}
> // ensure we have partitions assigned if we expect to
> if (subscriptions.partitionsAutoAssigned())
> coordinator.ensurePartitionAssignment();
> {noformat}
> I'm proposing to fix this by including same {{ensurePartitionAssignment}} 
> fragment in {{KafkaConsumer.subscriptions}} accessor as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {{KafkaConsumer}}, but these metrics put 
the {{TOPIC-PARTITION_ID}} inside of the metric name itself. These metrics 
should instead utilize the tags and put {{key="topic-partition"}} and 
{{value="TOPIC-PARTITION_ID"}}. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
{{key="node/topic"}} and {{value="NODE_ID/TOPIC_NAME"{}}}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to {{KafkaConsumer}}, but these metrics 
> put the {{TOPIC-PARTITION_ID}} inside of the metric name itself. These 
> metrics should instead utilize the tags and put {{key="topic-partition"}} and 
> {{value="TOPIC-PARTITION_ID"}}. 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> {{key="node/topic"}} and {{value="NODE_ID/TOPIC_NAME"{}}}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to 'KafkaConsumer', but these metrics put 
the TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to KafkaConsumer, but these metrics put 
> the TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
> instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to 'KafkaConsumer', but these metrics put 
the TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to 'KafkaConsumer', but these metrics put 
> the TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
> instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to ```KafkaConsumer```, but these metrics 
put the TOPIC-PARTITION_ID inside of the metric name itself. These metrics 
should instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to ```KafkaConsumer```, but these metrics 
> put the TOPIC-PARTITION_ID inside of the metric name itself. These metrics 
> should instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to ```KafkaConsumer```, but these metrics 
put the TOPIC-PARTITION_ID inside of the metric name itself. These metrics 
should instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to KafkaConsumer, but these metrics put 
> the TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
> instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {code}KafkaConsumer{code}, but these 
metrics put the TOPIC-PARTITION_ID inside of the metric name itself. These 
metrics should instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to {code}KafkaConsumer{code}, but these 
> metrics put the TOPIC-PARTITION_ID inside of the metric name itself. These 
> metrics should instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {code}KafkaConsumer{code}, but these 
metrics put the TOPIC-PARTITION_ID inside of the metric name itself. These 
metrics should instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to KafkaConsumer, but these metrics put 
> the TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
> instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {noformat}KafkaConsumer{noformat}, but 
these metrics put the {noformat}TOPIC-PARTITION_ID{noformat} inside of the 
metric name itself. These metrics should instead utilize the tags and put 
{noformat}key="topic-partition"{noformat} and 
{noformat}value="TOPIC-PARTITION_ID"{noformat}. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
{noformat}key="node/topic"{noformat} and 
{noformat}value="NODE_ID/TOPIC_NAME"{noformat}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to KafkaConsumer, but these metrics put 
> the TOPIC-PARTITION_ID inside of the metric name itself. These metrics should 
> instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {noformat}KafkaConsumer{noformat}, but 
these metrics put the {noformat}TOPIC-PARTITION_ID{noformat} inside of the 
metric name itself. These metrics should instead utilize the tags and put 
{noformat}key="topic-partition"{noformat} and 
{noformat}value="TOPIC-PARTITION_ID"{noformat}. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
{noformat}key="node/topic"{noformat} and 
{noformat}value="NODE_ID/TOPIC_NAME"{noformat}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {}KafkaConsumer{}, but these metrics put 
the {}TOPIC-PARTITION_ID{} inside of the metric name itself. These metrics 
should instead utilize the tags and put {}key="topic-partition"{} and 
{}value="TOPIC-PARTITION_ID"{}. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
{}key="node/topic"{} and {}value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to {noformat}KafkaConsumer{noformat}, but 
> these metrics put the {noformat}TOPIC-PARTITION_ID{noformat} inside of the 
> metric name itself. These metrics should instead utilize the tags and put 
> {noformat}key="topic-partition"{noformat} and 
> {noformat}value="TOPIC-PARTITION_ID"{noformat}. 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> {noformat}key="node/topic"{noformat} and 
> {noformat}value="NODE_ID/TOPIC_NAME"{noformat}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {}KafkaConsumer{}, but these metrics put 
the {}TOPIC-PARTITION_ID{} inside of the metric name itself. These metrics 
should instead utilize the tags and put {}key="topic-partition"{} and 
{}value="TOPIC-PARTITION_ID"{}. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
{}key="node/topic"{} and {}value="NODE_ID/TOPIC_NAME"{}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {KafkaConsumer}, but these metrics put 
the {TOPIC-PARTITION_ID} inside of the metric name itself. These metrics should 
instead utilize the tags and put {key="topic-partition"} and 
{value="TOPIC-PARTITION_ID"}. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
{key="node/topic"} and {value="NODE_ID/TOPIC_NAME"}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to {}KafkaConsumer{}, but these metrics 
> put the {}TOPIC-PARTITION_ID{} inside of the metric name itself. These 
> metrics should instead utilize the tags and put {}key="topic-partition"{} and 
> {}value="TOPIC-PARTITION_ID"{}. 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> {}key="node/topic"{} and {}value="NODE_ID/TOPIC_NAME"{}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to {KafkaConsumer}, but these metrics put 
the {TOPIC-PARTITION_ID} inside of the metric name itself. These metrics should 
instead utilize the tags and put {key="topic-partition"} and 
{value="TOPIC-PARTITION_ID"}. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
{key="node/topic"} and {value="NODE_ID/TOPIC_NAME"}.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to `KafkaConsumer`, but these metrics put 
the `TOPIC-PARTITION_ID` inside of the metric name itself. These metrics should 
instead utilize the tags and put `key="topic-partition"` and 
`value="TOPIC-PARTITION_ID"`. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
`key="node/topic"` and `value="NODE_ID/TOPIC_NAME"`.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to {KafkaConsumer}, but these metrics put 
> the {TOPIC-PARTITION_ID} inside of the metric name itself. These metrics 
> should instead utilize the tags and put {key="topic-partition"} and 
> {value="TOPIC-PARTITION_ID"}. 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> {key="node/topic"} and {value="NODE_ID/TOPIC_NAME"}.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to `KafkaConsumer`, but these metrics put 
the `TOPIC-PARTITION_ID` inside of the metric name itself. These metrics should 
instead utilize the tags and put `key="topic-partition"` and 
`value="TOPIC-PARTITION_ID"`. 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
`key="node/topic"` and `value="NODE_ID/TOPIC_NAME"`.

I'm guessing a KIP is needed for this as this changes a public API (metric)? 

  was:
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
"TOPIC-PARTITION_ID" inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME".


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to `KafkaConsumer`, but these metrics put 
> the `TOPIC-PARTITION_ID` inside of the metric name itself. These metrics 
> should instead utilize the tags and put `key="topic-partition"` and 
> `value="TOPIC-PARTITION_ID"`. 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> `key="node/topic"` and `value="NODE_ID/TOPIC_NAME"`.
> I'm guessing a KIP is needed for this as this changes a public API (metric)? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

Kevin Lu created KAFKA-5991:
---

 Summary: Change Consumer per partition lag metrics to put 
topic-partition-id in tags instead of metric name
 Key: KAFKA-5991
 URL: https://issues.apache.org/jira/browse/KAFKA-5991
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.11.0.0, 0.10.2.0, 0.10.1.0, 0.10.0.1, 0.10.0.0
Reporter: Kevin Lu
Priority: Minor


A KIP will be needed for this (?) as this requires a change to the public API 
(metric).

[KIP-92](https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer)
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
"TOPIC-PARTITION_ID" inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5991) Change Consumer per partition lag metrics to put topic-partition-id in tags instead of metric name

[Kevin Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lu updated KAFKA-5991:

Description: 
[KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
"TOPIC-PARTITION_ID" inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME".

  was:
A KIP will be needed for this (?) as this requires a change to the public API 
(metric).

[KIP-92](https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer)
 brought per partition lag metrics to KafkaConsumer, but these metrics put the 
"TOPIC-PARTITION_ID" inside of the metric name itself. These metrics should 
instead utilize the tags and put key="topic-partition" and 
value="TOPIC-PARTITION_ID". 

Per-broker (node) and per-topic metrics utilize tags in this way by putting 
key="node/topic" and value="NODE_ID/TOPIC_NAME".


> Change Consumer per partition lag metrics to put topic-partition-id in tags 
> instead of metric name
> --
>
> Key: KAFKA-5991
> URL: https://issues.apache.org/jira/browse/KAFKA-5991
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.2.0, 0.11.0.0
>Reporter: Kevin Lu
>Priority: Minor
>  Labels: metrics
>
> [KIP-92|https://cwiki.apache.org/confluence/display/KAFKA/KIP-92+-+Add+per+partition+lag+metrics+to+KafkaConsumer]
>  brought per partition lag metrics to KafkaConsumer, but these metrics put 
> the "TOPIC-PARTITION_ID" inside of the metric name itself. These metrics 
> should instead utilize the tags and put key="topic-partition" and 
> value="TOPIC-PARTITION_ID". 
> Per-broker (node) and per-topic metrics utilize tags in this way by putting 
> key="node/topic" and value="NODE_ID/TOPIC_NAME".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5990) Add generated documentation for Connect metrics

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-5990:


 Summary: Add generated documentation for Connect metrics
 Key: KAFKA-5990
 URL: https://issues.apache.org/jira/browse/KAFKA-5990
 Project: Kafka
  Issue Type: Sub-task
  Components: documentation, KafkaConnect
Affects Versions: 0.11.0.0
Reporter: Randall Hauch
Assignee: Randall Hauch
 Fix For: 1.0.0


KAFKA-5191 recently added a new {{MetricNameTemplate}} that is used to create 
the {{MetricName}} objects in the producer and consumer, as we as in the 
newly-added generation of metric documentation. The {{Metric.toHtmlTable}} 
method then takes these templates and generates an HTML documentation for the 
metrics.

Change the Connect metrics to use these templates and update the build to 
generate these metrics and include them in the Kafka documentation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5989) disableLogging() causes an initialization loop

[Tuan Nguyen (JIRA)]

Tuan Nguyen created KAFKA-5989:
--

 Summary: disableLogging() causes an initialization loop
 Key: KAFKA-5989
 URL: https://issues.apache.org/jira/browse/KAFKA-5989
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.1
Reporter: Tuan Nguyen
 Attachments: App.java

Using {{disableLogging()}} for either of the built-in state store types causes 
an initialization loop in the StreamThread.

Case A - this works just fine:
{code}
final StateStoreSupplier testStore = Stores.create(topic)
.withStringKeys()
.withStringValues()
.inMemory()
//  .disableLogging() 
.maxEntries(10)
.build();
{code}

Case B - this does not:
{code}
final StateStoreSupplier testStore = Stores.create(topic)
.withStringKeys()
.withStringValues()
.inMemory()
.disableLogging() 
.maxEntries(10)
.build();
{code}

A brief debugging dive shows that in Case B, 
{{AssignedTasks.allTasksRunning()}} never returns true, because of a remnant 
entry in {{AssignedTasks#restoring}} that never gets properly restored.

See [^App.java] for a working test (requires ZK + Kafka ensemble, and at least 
one keyed message produced to the "test" topic)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5988) Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE

[Ted Yu (JIRA)]

Ted Yu created KAFKA-5988:
-

 Summary: Consider removing StreamThread#STREAM_THREAD_ID_SEQUENCE
 Key: KAFKA-5988
 URL: https://issues.apache.org/jira/browse/KAFKA-5988
 Project: Kafka
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor


StreamThread#STREAM_THREAD_ID_SEQUENCE is used for naming (numbering) 
StreamThread's .
It is used in create() which is called from a loop in KafkaStreams ctor.

We can remove STREAM_THREAD_ID_SEQUENCE and pass the loop index to create()



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5746) Add new metrics to support health checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184866#comment-16184866
 ] 

ASF GitHub Bot commented on KAFKA-5746:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3705


> Add new metrics to support health checks
> 
>
> Key: KAFKA-5746
> URL: https://issues.apache.org/jira/browse/KAFKA-5746
> Project: Kafka
>  Issue Type: New Feature
>  Components: metrics
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 1.0.0
>
>
> It will be useful to have some additional metrics to support health checks.
> Details are in 
> [KIP-188|https://cwiki.apache.org/confluence/display/KAFKA/KIP-188+-+Add+new+metrics+to+support+health+checks]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5949) User Callback Exceptions need to be handled properly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184864#comment-16184864
 ] 

ASF GitHub Bot commented on KAFKA-5949:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/3986

KAFKA-5949: Follow-up after latest KIP-161 changes

 - compare KAFKA-5958

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-5949-exceptions-user-callbacks-KIP-161-follow-up

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3986.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3986


commit 95ebc1ef168b35fe3c7e1e23201114fbafee12bb
Author: Matthias J. Sax 
Date:   2017-09-28T20:58:45Z

KAFKA-5949: Follow-up after latest KIP-161 changes
 - compare KAFKA-5958




> User Callback Exceptions need to be handled properly
> 
>
> Key: KAFKA-5949
> URL: https://issues.apache.org/jira/browse/KAFKA-5949
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 1.0.0
>
>
> In Streams, we allow to register multiple user callbacks. We need to handle 
> those exceptions gracefully, by catching and wrapping with a StreamsException.
> - TimestampExtractor
> - DeserializationHandler
> - StateRestoreListener



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5552) testTransactionalProducerTopicAuthorizationExceptionInCommit fails

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta resolved KAFKA-5552.
-
   Resolution: Cannot Reproduce
Fix Version/s: (was: 1.1.0)

> testTransactionalProducerTopicAuthorizationExceptionInCommit fails 
> ---
>
> Key: KAFKA-5552
> URL: https://issues.apache.org/jira/browse/KAFKA-5552
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Eno Thereska
>
> Got a unit test error: 
> https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5877/
> Error Message
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
> Stacktrace
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:524)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.beginCommit(TransactionManager.java:190)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.commitTransaction(KafkaProducer.java:583)
>   at 
> kafka.api.AuthorizerIntegrationTest.testTransactionalProducerTopicAuthorizationExceptionInCommit(AuthorizerIntegrationTest.scala:1027)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> 

[jira] [Commented] (KAFKA-5552) testTransactionalProducerTopicAuthorizationExceptionInCommit fails

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184863#comment-16184863
 ] 

Apurva Mehta commented on KAFKA-5552:
-

I have not seen this test fail in ages, and can probably be closed.

> testTransactionalProducerTopicAuthorizationExceptionInCommit fails 
> ---
>
> Key: KAFKA-5552
> URL: https://issues.apache.org/jira/browse/KAFKA-5552
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Eno Thereska
>
> Got a unit test error: 
> https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5877/
> Error Message
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
> Stacktrace
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:524)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.beginCommit(TransactionManager.java:190)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.commitTransaction(KafkaProducer.java:583)
>   at 
> kafka.api.AuthorizerIntegrationTest.testTransactionalProducerTopicAuthorizationExceptionInCommit(AuthorizerIntegrationTest.scala:1027)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> 

[jira] [Commented] (KAFKA-5870) Idempotent producer: a producerId reset causes undesirable behavior for inflight batches to other partitions

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184862#comment-16184862
 ] 

Apurva Mehta commented on KAFKA-5870:
-

The only downside is that when previously sent batches expire, _and_ inflight 
batches get a retriable error, then the latter will get a spurious 
{{OutOfOrderSequenceException}}.

I think we should fix it, but the 'proper' fix would require yet more state in 
the producer: reset is requested, but should only kick in once there are no 
more inflights, and there are no more batches with the old producer id in the 
queue (ie. those which were previously drained and are now being retried). And 
how long should we wait to drain the old batches in the queue and block 
everything else (which may cause more batches to expire, etc.). 

There are a few things think through. So I am hesitant to add that complication 
to 1.0.0, since we already have a major complication with max.in.flight > 1.

Perhaps a simple solution is to send a different exception when we give up on 
the inflight batch, rather than an {{OutOfOrderSequenceException}}, to reduce 
confusion. Maybe an {{OperationNotAttemptedException}} ?

> Idempotent producer: a producerId reset causes undesirable behavior for 
> inflight batches to other partitions
> 
>
> Key: KAFKA-5870
> URL: https://issues.apache.org/jira/browse/KAFKA-5870
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>  Labels: exactly-once
> Fix For: 1.1.0
>
>
> Currently, if we have to reset the producer id for any reason (for instance 
> if batches to a partition get expired, if we get an 
> {{OutOfOrderSequenceException}}, etc) we could cause batches to other 
> --healthy-- partitions to fail with a spurious 
> {{OutOfOrderSequenceException}}.
> This is detailed in this PR discussion: 
> https://github.com/apache/kafka/pull/3743#discussion_r137907630
> Ideally, we would want all inflight batches to be handled to completion 
> rather than potentially failing them prematurely. Further, since we want to 
> tighten up the semantics of the {{OutOfOrderSequenceException}}, at the very 
> least we should raise another exception in this case, because there is no 
> data loss on the broker when the client gives up. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5562) Do streams state directory cleanup on a single thread

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5562:
---
Fix Version/s: 0.11.0.1

> Do streams state directory cleanup on a single thread
> -
>
> Key: KAFKA-5562
> URL: https://issues.apache.org/jira/browse/KAFKA-5562
> Project: Kafka
>  Issue Type: Bug
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.11.0.1
>
>
> Currently in streams we clean up old state directories every so often (as 
> defined by {{state.cleanup.delay.ms}}). However, every StreamThread runs the 
> cleanup, which is both unnecessary and can potentially lead to race 
> conditions.
> It would be better to perform the state cleanup on a single thread and only 
> when the {{KafkaStreams}} instance is in a running state.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184797#comment-16184797
 ] 

Matthias J. Sax commented on KAFKA-5967:


[~sssanthalingam] Please send an email with you user name to Kafka dev list and 
ask to be added to contributors list. Than you can assign to yourself.

> Ineffective check of negative value in 
> CompositeReadOnlyKeyValueStore#approximateNumEntries()
> -
>
> Key: KAFKA-5967
> URL: https://issues.apache.org/jira/browse/KAFKA-5967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>  Labels: beginner, newbie
>
> {code}
> long total = 0;
> for (ReadOnlyKeyValueStore store : stores) {
> total += store.approximateNumEntries();
> }
> return total < 0 ? Long.MAX_VALUE : total;
> {code}
> The check for negative value seems to account for wrapping.
> However, wrapping can happen within the for loop. So the check should be 
> performed inside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5987) Kafka metrics templates used in document generation should maintain order of tags

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184796#comment-16184796
 ] 

ASF GitHub Bot commented on KAFKA-5987:
---

GitHub user rhauch opened a pull request:

https://github.com/apache/kafka/pull/3985

KAFKA-5987: Maintain order of metric tags in generated documentation

The `MetricNameTemplate` is changed to used a `LinkedHashSet` to maintain 
the same order of the tags that are passed in. This tag order is then 
maintained when `Metrics.toHtmlTable` generates the MBean names for each of the 
metrics.

The `SenderMetricsRegistry` and `FetcherMetricsRegistry` both contain 
templates used in the producer and consumer, respectively, and these were 
changed to use a `LinkedHashSet` to maintain the order of the tags.

Before this change, the generated HTML documentation might use MBean names 
like the following and order them:

```

kafka.connect:type=sink-task-metrics,connector={connector},partition={partition},task={task},topic={topic}
kafka.connect:type=sink-task-metrics,connector={connector},task={task}
```
However, after this change, the documentation would use the following order:
```
kafka.connect:type=sink-task-metrics,connector={connector},task={task}

kafka.connect:type=sink-task-metrics,connector={connector},task={task},topic={topic},partition={partition}
```

This is more readable as the code that is creating the templates has 
control over the order of the tags.

Note that JMX MBean names use ObjectName that does not maintain order of 
the properties (tags), so this change should have no impact on the actual JMX 
MBean names used in the metrics.

cc @wushujames

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rhauch/kafka kafka-5987

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3985.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3985


commit e5379c999b03b1a10d3779e21659f4a7808dc53c
Author: Randall Hauch 
Date:   2017-09-28T20:07:34Z

KAFKA-5987 Maintain order of metric tags in generated documentation

The `MetricNameTemplate` is changed to used a `LinkedHashSet` to maintain 
the same order of the tags that are passed in. This tag order is then 
maintained when `Metrics.toHtmlTable` generates the MBean names for each of the 
metrics.

The `SenderMetricsRegistry` and `FetcherMetricsRegistry` both contain 
templates used in the producer and consumer, respectively, and these were 
changed to use a `LinkedHashSet` to maintain the order of the tags.




> Kafka metrics templates used in document generation should maintain order of 
> tags
> -
>
> Key: KAFKA-5987
> URL: https://issues.apache.org/jira/browse/KAFKA-5987
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 1.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
> Fix For: 1.0.0
>
>
> KAFKA-5191 recently added a new {{MetricNameTemplate}} that is used to create 
> the {{MetricName}} objects in the producer and consumer, as we as in the 
> newly-added generation of metric documentation. The {{MetricNameTemplate}} 
> and the {{Metric.toHtmlTable}} do not maintain the order of the tags, which 
> means the resulting HTML documentation will order the table of MBean 
> attributes based upon the lexicographical ordering of the MBeans, each of 
> which uses the lexicographical ordering of its tags. This can result in the 
> following order:
> {noformat}
> kafka.connect:type=sink-task-metrics,connector={connector},partition={partition},task={task},topic={topic}
> kafka.connect:type=sink-task-metrics,connector={connector},task={task}
> {noformat}
> However, if the MBeans maintained the order of the tags then the 
> documentation would use the following order:
> {noformat}
> kafka.connect:type=sink-task-metrics,connector={connector},task={task}
> kafka.connect:type=sink-task-metrics,connector={connector},task={task},topic={topic},partition={partition}
> {noformat}
> This would be more readable, and the code that is creating the templates 
> would have control over the order of the tags. 
> To maintain order, {{MetricNameTemplate}} should used a {{LinkedHashSet}} for 
> the tags, and the {{Metrics.toHtmlTable}} method should also use a 
> {{LinkedHashMap}} when building up the tags used in the MBean name.
> Note that JMX MBean names use {{ObjectName}} that does not maintain order, so 
> this change should have no impact on JMX MBean names.



--
This message was 

[jira] [Commented] (KAFKA-5888) Transactions system test should check for message order

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184769#comment-16184769
 ] 

ASF GitHub Bot commented on KAFKA-5888:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3969


> Transactions system test should check for message order
> ---
>
> Key: KAFKA-5888
> URL: https://issues.apache.org/jira/browse/KAFKA-5888
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>  Labels: exactly-once
> Fix For: 1.0.0
>
>
> Currently, the transactions system test doesn't check for correct ordering of 
> the messages in a transaction. With KAFKA-5494, we can have multiple inflight 
> requests for a single transaction, which could yield to out of order messages 
> in the log if there are bugs. So we should assert that order is maintained in 
> our system tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5987) Kafka metrics templates used in document generation should maintain order of tags

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-5987:


 Summary: Kafka metrics templates used in document generation 
should maintain order of tags
 Key: KAFKA-5987
 URL: https://issues.apache.org/jira/browse/KAFKA-5987
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 1.0.0
Reporter: Randall Hauch
Assignee: Randall Hauch
 Fix For: 1.0.0


KAFKA-5191 recently added a new {{MetricNameTemplate}} that is used to create 
the {{MetricName}} objects in the producer and consumer, as we as in the 
newly-added generation of metric documentation. The {{MetricNameTemplate}} and 
the {{Metric.toHtmlTable}} do not maintain the order of the tags, which means 
the resulting HTML documentation will order the table of MBean attributes based 
upon the lexicographical ordering of the MBeans, each of which uses the 
lexicographical ordering of its tags. This can result in the following order:

{noformat}
kafka.connect:type=sink-task-metrics,connector={connector},partition={partition},task={task},topic={topic}
kafka.connect:type=sink-task-metrics,connector={connector},task={task}
{noformat}

However, if the MBeans maintained the order of the tags then the documentation 
would use the following order:

{noformat}
kafka.connect:type=sink-task-metrics,connector={connector},task={task}
kafka.connect:type=sink-task-metrics,connector={connector},task={task},topic={topic},partition={partition}
{noformat}

This would be more readable, and the code that is creating the templates would 
have control over the order of the tags. 

To maintain order, {{MetricNameTemplate}} should used a {{LinkedHashSet}} for 
the tags, and the {{Metrics.toHtmlTable}} method should also use a 
{{LinkedHashMap}} when building up the tags used in the MBean name.

Note that JMX MBean names use {{ObjectName}} that does not maintain order, so 
this change should have no impact on JMX MBean names.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5473) handle ZK session expiration properly when a new session can't be established

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184683#comment-16184683
 ] 

Ismael Juma commented on KAFKA-5473:


We would need the PR to be submitted ASAP so that there's time for review 
though. If your availability is restricted, would you be OK with someone else 
picking it up? This can cause some very bad behaviour, so I'm eager to get it 
fixed for 1.0.0.

> handle ZK session expiration properly when a new session can't be established
> -
>
> Key: KAFKA-5473
> URL: https://issues.apache.org/jira/browse/KAFKA-5473
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Prasanna Gautam
> Fix For: 1.0.0
>
>
> In https://issues.apache.org/jira/browse/KAFKA-2405, we change the logic in 
> handling ZK session expiration a bit. If a new ZK session can't be 
> established after session expiration, we just log an error and continue. 
> However, this can leave the broker in a bad state since it's up, but not 
> registered from the controller's perspective. Replicas on this broker may 
> never to be in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5473) handle ZK session expiration properly when a new session can't be established

[Prasanna Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184653#comment-16184653
 ] 

Prasanna Gautam commented on KAFKA-5473:


Think I can make it to the code freeze. I'm at a conference this week and it's 
a bit of hassle to get the env well setup on the machine I have here. Is there 
an easy way to bootstrap the environment for testing? I'd like to reuse 
anything that's already been done for that.

> handle ZK session expiration properly when a new session can't be established
> -
>
> Key: KAFKA-5473
> URL: https://issues.apache.org/jira/browse/KAFKA-5473
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Prasanna Gautam
> Fix For: 1.0.0
>
>
> In https://issues.apache.org/jira/browse/KAFKA-2405, we change the logic in 
> handling ZK session expiration a bit. If a new ZK session can't be 
> established after session expiration, we just log an error and continue. 
> However, this can leave the broker in a bad state since it's up, but not 
> registered from the controller's perspective. Replicas on this broker may 
> never to be in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184644#comment-16184644
 ] 

Ted Yu commented on KAFKA-5967:
---

Once you send out pull request, a committer would assign to you.

> Ineffective check of negative value in 
> CompositeReadOnlyKeyValueStore#approximateNumEntries()
> -
>
> Key: KAFKA-5967
> URL: https://issues.apache.org/jira/browse/KAFKA-5967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>  Labels: beginner, newbie
>
> {code}
> long total = 0;
> for (ReadOnlyKeyValueStore store : stores) {
> total += store.approximateNumEntries();
> }
> return total < 0 ? Long.MAX_VALUE : total;
> {code}
> The check for negative value seems to account for wrapping.
> However, wrapping can happen within the for loop. So the check should be 
> performed inside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()

[siva santhalingam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184626#comment-16184626
 ] 

siva santhalingam commented on KAFKA-5967:
--

[~te...@apache.org] Could you please assign this to me, looks like i dont have 
access to assign this to myself.

> Ineffective check of negative value in 
> CompositeReadOnlyKeyValueStore#approximateNumEntries()
> -
>
> Key: KAFKA-5967
> URL: https://issues.apache.org/jira/browse/KAFKA-5967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>  Labels: beginner, newbie
>
> {code}
> long total = 0;
> for (ReadOnlyKeyValueStore store : stores) {
> total += store.approximateNumEntries();
> }
> return total < 0 ? Long.MAX_VALUE : total;
> {code}
> The check for negative value seems to account for wrapping.
> However, wrapping can happen within the for loop. So the check should be 
> performed inside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5967) Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries()

[siva santhalingam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184607#comment-16184607
 ] 

siva santhalingam commented on KAFKA-5967:
--

[~te...@apache.org] Can i assign this to myself? 

> Ineffective check of negative value in 
> CompositeReadOnlyKeyValueStore#approximateNumEntries()
> -
>
> Key: KAFKA-5967
> URL: https://issues.apache.org/jira/browse/KAFKA-5967
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>  Labels: beginner, newbie
>
> {code}
> long total = 0;
> for (ReadOnlyKeyValueStore store : stores) {
> total += store.approximateNumEntries();
> }
> return total < 0 ? Long.MAX_VALUE : total;
> {code}
> The check for negative value seems to account for wrapping.
> However, wrapping can happen within the for loop. So the check should be 
> performed inside the loop.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5473) handle ZK session expiration properly when a new session can't be established

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5473:
---
Fix Version/s: (was: 1.0.1)
   1.0.0

> handle ZK session expiration properly when a new session can't be established
> -
>
> Key: KAFKA-5473
> URL: https://issues.apache.org/jira/browse/KAFKA-5473
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Prasanna Gautam
> Fix For: 1.0.0
>
>
> In https://issues.apache.org/jira/browse/KAFKA-2405, we change the logic in 
> handling ZK session expiration a bit. If a new ZK session can't be 
> established after session expiration, we just log an error and continue. 
> However, this can leave the broker in a bad state since it's up, but not 
> registered from the controller's perspective. Replicas on this broker may 
> never to be in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5473) handle ZK session expiration properly when a new session can't be established

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184496#comment-16184496
 ] 

Ismael Juma commented on KAFKA-5473:


Do you think you can submit it within the next few days? The code freeze for 
1.0.0 is next Wednesday and it would be good to get this bug fix in before 
then. Let us know if you need some help.

> handle ZK session expiration properly when a new session can't be established
> -
>
> Key: KAFKA-5473
> URL: https://issues.apache.org/jira/browse/KAFKA-5473
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Prasanna Gautam
> Fix For: 1.0.0
>
>
> In https://issues.apache.org/jira/browse/KAFKA-2405, we change the logic in 
> handling ZK session expiration a bit. If a new ZK session can't be 
> established after session expiration, we just log an error and continue. 
> However, this can leave the broker in a bad state since it's up, but not 
> registered from the controller's perspective. Replicas on this broker may 
> never to be in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5985) Mention the need to close store iterators

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5985:
---
Component/s: streams

> Mention the need to close store iterators
> -
>
> Key: KAFKA-5985
> URL: https://issues.apache.org/jira/browse/KAFKA-5985
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation, streams
>Affects Versions: 0.11.0.0
>Reporter: Stanislav Chizhov
>
> Store iterators should be closed in all/most of the cases, but currently it 
> is not consistently reflected in the documentation and javadocs. For instance 
>  
> https://kafka.apache.org/0110/documentation/streams/developer-guide#streams_developer-guide_interactive-queries_custom-stores
>  does not mention the need to close an iterator and provide an example that 
> does not do that. 
> Some of the fetch methods do mention the need to close an iterator returned 
> (e.g. 
> https://kafka.apache.org/0110/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#range(K,%20K)),
>  but others do not: 
> https://kafka.apache.org/0110/javadoc/org/apache/kafka/streams/state/ReadOnlyWindowStore.html#fetch(K,%20long,%20long)
> It makes sense to: 
> - update javadoc for all store methods that do return iterators to reflect 
> that the iterator returned needs to be closed
> - mention it in the documentation and to update related examples.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5901) Create Connect metrics for source tasks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184453#comment-16184453
 ] 

ASF GitHub Bot commented on KAFKA-5901:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3959


> Create Connect metrics for source tasks
> ---
>
> Key: KAFKA-5901
> URL: https://issues.apache.org/jira/browse/KAFKA-5901
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Blocker
> Fix For: 1.0.0
>
>
> See KAFKA-2376 for parent task and 
> [KIP-196|https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework]
>  for the details on the metrics. This subtask is to create the "Source Task 
> Metrics".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5156) Options for handling exceptions in streams

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5156:
---
Fix Version/s: (was: 1.0.0)

> Options for handling exceptions in streams
> --
>
> Key: KAFKA-5156
> URL: https://issues.apache.org/jira/browse/KAFKA-5156
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Matthias J. Sax
>  Labels: user-experience
>
> This is a task around options for handling exceptions in streams. It focuses 
> around options for dealing with corrupt data (keep going, stop streams, log, 
> retry, etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5158) Options for handling exceptions during processing

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5158:
---
Fix Version/s: (was: 1.0.0)

> Options for handling exceptions during processing
> -
>
> Key: KAFKA-5158
> URL: https://issues.apache.org/jira/browse/KAFKA-5158
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Eno Thereska
>Assignee: Matthias J. Sax
>
> Imagine the app-level processing of a (non-corrupted) record fails (e.g. the 
> user attempted to do a RPC to an external system, and this call failed). How 
> can you process such failed records in a scalable way? For example, imagine 
> you need to implement a retry policy such as "retry with exponential 
> backoff". Here, you have the problem that 1. you can't really pause 
> processing a single record because this will pause the processing of the full 
> stream (bottleneck!) and 2. there is no straight-forward way to "sort" failed 
> records based on their "next retry time".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5217) Improve Streams internal exception handling

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5217:
---
Fix Version/s: (was: 1.0.0)

> Improve Streams internal exception handling
> ---
>
> Key: KAFKA-5217
> URL: https://issues.apache.org/jira/browse/KAFKA-5217
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>
> Streams does not handle all exceptions gracefully atm, but tend to throw 
> exceptions to the user, even if we could handle them internally and recover 
> automatically. We want to revisit this exception handling to be more 
> resilient.
> For example, for any kind of rebalance exception, we should just log it, and 
> rejoin the consumer group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5302) Improve exception handling on streams client (communication with brokers)

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5302:
---
Fix Version/s: (was: 1.0.0)

> Improve exception handling on streams client (communication with brokers)
> -
>
> Key: KAFKA-5302
> URL: https://issues.apache.org/jira/browse/KAFKA-5302
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Eno Thereska
>Assignee: Matthias J. Sax
>
> These are exceptions in StreamsKafkaClient.java.
> Currently throws either StreamsException or BrokerNotFoundException.
> Used by InternalTopicManager to create topics and get their metadata.
> Used by StreamPartitionAssignor. 
> Currently InternalTopicManager retries a few times after catching an 
> exception. 
> A failure here is sent all the way up to the stream thread and will stop the 
> streams pipeline. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (KAFKA-5859) Avoid retaining AbstractRequest in RequestChannel.Request

[Sean Glover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Glover reassigned KAFKA-5859:
--

Assignee: Sean Glover

> Avoid retaining AbstractRequest in RequestChannel.Request
> -
>
> Key: KAFKA-5859
> URL: https://issues.apache.org/jira/browse/KAFKA-5859
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Sean Glover
>Priority: Minor
>  Labels: newbie
>
> We currently store AbstractRequest in RequestChannel.Request.bodyAndSize. 
> RequestChannel.Request is, in turn, stored in RequestChannel.Response. We 
> keep the latter until the response is sent to the client.
> However, after KafkaApis.handle, we no longer need AbstractRequest apart from 
> its string representation for logging. We could potentially replace 
> AbstractRequest with a String representation (if the relevant logging is 
> enabled). The String representation is generally small while some 
> AbstractRequest subclasses can be pretty large. The largest one is 
> ProduceRequest and we clear the underlying ByteBuffer explicitly in 
> KafkaApis.handleProduceRequest. We could potentially remove that special case 
> if AbstractRequest subclasses were not retained.
> This was originally suggested by [~hachikuji] in the following PR 
> https://github.com/apache/kafka/pull/3801#discussion_r137592277



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[Mikhail Lesyk (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184386#comment-16184386
 ] 

Mikhail Lesyk commented on KAFKA-4477:
--

Updated from 0.10.2.1 to 0.11.0.1 and started hitting this issue.
Only restart of node helps.

> Node reduces its ISR to itself, and doesn't recover. Other nodes do not take 
> leadership, cluster remains sick until node is restarted.
> --
>
> Key: KAFKA-4477
> URL: https://issues.apache.org/jira/browse/KAFKA-4477
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
> Environment: RHEL7
> java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
>Reporter: Michael Andre Pearce
>Assignee: Apurva Mehta
>Priority: Critical
>  Labels: reliability
> Fix For: 0.10.1.1
>
> Attachments: 2016_12_15.zip, 72_Server_Thread_Dump.txt, 
> 73_Server_Thread_Dump.txt, 74_Server_Thread_Dump, issue_node_1001_ext.log, 
> issue_node_1001.log, issue_node_1002_ext.log, issue_node_1002.log, 
> issue_node_1003_ext.log, issue_node_1003.log, kafka.jstack, 
> server_1_72server.log, server_2_73_server.log, server_3_74Server.log, 
> state_change_controller.tar.gz
>
>
> We have encountered a critical issue that has re-occured in different 
> physical environments. We haven't worked out what is going on. We do though 
> have a nasty work around to keep service alive. 
> We do have not had this issue on clusters still running 0.9.01.
> We have noticed a node randomly shrinking for the partitions it owns the 
> ISR's down to itself, moments later we see other nodes having disconnects, 
> followed by finally app issues, where producing to these partitions is 
> blocked.
> It seems only by restarting the kafka instance java process resolves the 
> issues.
> We have had this occur multiple times and from all network and machine 
> monitoring the machine never left the network, or had any other glitches.
> Below are seen logs from the issue.
> Node 7:
> [2016-12-01 07:01:28,112] INFO Partition 
> [com_ig_trade_v1_position_event--demo--compacted,10] on broker 7: Shrinking 
> ISR for partition [com_ig_trade_v1_position_event--demo--compacted,10] from 
> 1,2,7 to 7 (kafka.cluster.Partition)
> All other nodes:
> [2016-12-01 07:01:38,172] WARN [ReplicaFetcherThread-0-7], Error in fetch 
> kafka.server.ReplicaFetcherThread$FetchRequest@5aae6d42 
> (kafka.server.ReplicaFetcherThread)
> java.io.IOException: Connection to 7 was disconnected before the response was 
> read
> All clients:
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> After this occurs, we then suddenly see on the sick machine an increasing 
> amount of close_waits and file descriptors.
> As a work around to keep service we are currently putting in an automated 
> process that tails and regex's for: and where new_partitions hit just itself 
> we restart the node. 
> "\[(?P.+)\] INFO Partition \[.*\] on broker .* Shrinking ISR for 
> partition \[.*\] from (?P.+) to (?P.+) 
> \(kafka.cluster.Partition\)"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5473) handle ZK session expiration properly when a new session can't be established

[Prasanna Gautam (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184314#comment-16184314
 ] 

Prasanna Gautam commented on KAFKA-5473:


[~ijuma] Yes I intend to send a PR for this. I need to resume this and test. 

> handle ZK session expiration properly when a new session can't be established
> -
>
> Key: KAFKA-5473
> URL: https://issues.apache.org/jira/browse/KAFKA-5473
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Prasanna Gautam
> Fix For: 1.0.1
>
>
> In https://issues.apache.org/jira/browse/KAFKA-2405, we change the logic in 
> handling ZK session expiration a bit. If a new ZK session can't be 
> established after session expiration, we just log an error and continue. 
> However, this can leave the broker in a bad state since it's up, but not 
> registered from the controller's perspective. Replicas on this broker may 
> never to be in sync.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3999) Consumer bytes-fetched metric uses decompressed message size

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3999:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Consumer bytes-fetched metric uses decompressed message size
> 
>
> Key: KAFKA-3999
> URL: https://issues.apache.org/jira/browse/KAFKA-3999
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>Priority: Minor
> Fix For: 1.1.0
>
>
> It looks like the computation for the bytes-fetched metrics uses the size of 
> the decompressed message set. I would have expected it to be based off of the 
> raw size of the fetch responses. Perhaps it would be helpful to expose both 
> the raw and decompressed fetch sizes? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2394:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Use RollingFileAppender by default in log4j.properties
> --
>
> Key: KAFKA-2394
> URL: https://issues.apache.org/jira/browse/KAFKA-2394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Dustin Cote
>Priority: Minor
>  Labels: newbie
> Fix For: 1.1.0
>
> Attachments: log4j.properties.patch
>
>
> The default log4j.properties bundled with Kafka uses ConsoleAppender and 
> DailyRollingFileAppender, which offer no protection to users from spammy 
> logging. In extreme cases (such as when issues like KAFKA-1461 are 
> encountered), the logs can exhaust the local disk space. This could be a 
> problem for Kafka adoption since new users are less likely to adjust the 
> logging properties themselves, and are more likely to have configuration 
> problems which result in log spam. 
> To fix this, we can use RollingFileAppender, which offers two settings for 
> controlling the maximum space that log files will use.
> maxBackupIndex: how many backup files to retain
> maxFileSize: the max size of each log file
> One question is whether this change is a compatibility concern? The backup 
> strategy and filenames used by RollingFileAppender are different from those 
> used by DailyRollingFileAppender, so any tools which depend on the old format 
> will break. If we think this is a serious problem, one solution would be to 
> provide two versions of log4j.properties and add a flag to enable the new 
> one. Another solution would be to include the RollingFileAppender 
> configuration in the default log4j.properties, but commented out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5359) Exceptions from RequestFuture lack parts of the stack trace

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5359:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Exceptions from RequestFuture lack parts of the stack trace
> ---
>
> Key: KAFKA-5359
> URL: https://issues.apache.org/jira/browse/KAFKA-5359
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Magnus Reftel
>Assignee: Vahid Hashemian
>Priority: Minor
> Fix For: 1.1.0
>
>
> When an exception occurs within a task that reports its result using a 
> RequestFuture, that exception is stored in a field on the RequestFuture using 
> the {{raise}} method. In many places in the code where such futures are 
> completed, that exception is then thrown directly using {{throw 
> future.exception();}} (see e.g. 
> [Fetcher.getTopicMetadata|https://github.com/apache/kafka/blob/aebba89a2b9b5ea6a7cab2599555232ef3fe21ad/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L316]).
> This means that the exception that ends up in client code only has stack 
> traces related to the original exception, but nothing leading up to the 
> completion of the future. The client therefore gets no indication of what was 
> going on in the client code - only that it somehow ended up in the Kafka 
> libraries, and that a task failed at some point.
> One solution to this is to use the exceptions from the future as causes for 
> chained exceptions, so that the client gets a stack trace that shows what the 
> client was doing, in addition to getting the stack traces for the exception 
> in the task.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-4333) Report consumer group coordinator id when '--list' option is used

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4333:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Report consumer group coordinator id when '--list' option is used
> -
>
> Key: KAFKA-4333
> URL: https://issues.apache.org/jira/browse/KAFKA-4333
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
> Fix For: 1.1.0
>
>
> One piece of information missing when extracting information about consumer 
> groups (Java API based) is the coordinator id (broker id of the coordinator). 
> It would be useful to enhance the {{--list}} option of the consumer group 
> command to report the corresponding coordinator id of each consumer group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3297:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> More optimally balanced partition assignment strategy (new consumer)
> 
>
> Key: KAFKA-3297
> URL: https://issues.apache.org/jira/browse/KAFKA-3297
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Andrew Olson
>Assignee: Andrew Olson
> Fix For: 1.1.0
>
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the new consumer. For the original high-level consumer, 
> see KAFKA-2435.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5986) Streams State Restoration never completes when logging is disabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184060#comment-16184060
 ] 

ASF GitHub Bot commented on KAFKA-5986:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3983

KAFKA-5986: Streams State Restoration never completes when logging is 
disabled

When logging is disabled and there are state stores the task never 
transitions from restoring to running. This is because we only ever check if 
the task has state stores and return false on initialization if it does. The 
check should be if we have changelog partitions, i.e., we need to restore.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka restore-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3983.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3983


commit c60856362a9dfe1f8b68d76cbb5a783eef6abfff
Author: Damian Guy 
Date:   2017-09-28T11:52:34Z

fix task initialization when logging disabled




> Streams State Restoration never completes when logging is disabled
> --
>
> Key: KAFKA-5986
> URL: https://issues.apache.org/jira/browse/KAFKA-5986
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.1
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Critical
> Fix For: 1.0.0, 0.11.0.2
>
>
> When logging is disabled on a state store, the store restoration never 
> completes. This is likely because there are no changelogs, but more 
> investigation is required.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5276) Support derived and prefixed configs in DescribeConfigs (KIP-133)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5276:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Support derived and prefixed configs in DescribeConfigs (KIP-133)
> -
>
> Key: KAFKA-5276
> URL: https://issues.apache.org/jira/browse/KAFKA-5276
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 1.1.0
>
>
> The broker supports config overrides per listener. The way we do that is by 
> prefixing the configs with the listener name. These configs are not defined 
> by ConfigDef and they don't appear in `values()`. They do appear in 
> `originals()`. We should change the code so that we return these configs. 
> Because these configs are read-only, nothing needs to be done for 
> AlterConfigs.
> With regards to derived configs, an example is advertised.listeners, which 
> falls back to listeners. This is currently done outside AbstractConfig. We 
> should look into including these into AbstractConfig so that the fallback 
> happens for the returned configs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-4862) Kafka client connect to a shutdown node will block for a long time

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4862:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Kafka client connect to a shutdown node will block for a long time
> --
>
> Key: KAFKA-4862
> URL: https://issues.apache.org/jira/browse/KAFKA-4862
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0, 0.10.2.0
>Reporter: Pengwei
>Assignee: Pengwei
> Fix For: 1.1.0
>
>
> Currently in our test env, we found after one of the broker node crash(reboot 
> or os crash), the client maybe still connecting to the crash node to send 
> metadata request or other request, and it need about several  minutes to 
> aware the connection is timeout then try another node to connect to send the 
> request.  Then the client may still not aware the metadata change after 
> several minutes.
> We don't have a connection timeout for the network client, we should add a 
> connection timeout for the client



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4808:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> send of null key to a compacted topic should throw error back to user
> -
>
> Key: KAFKA-4808
> URL: https://issues.apache.org/jira/browse/KAFKA-4808
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.0
>Reporter: Ismael Juma
>Assignee: Mayuresh Gharat
> Fix For: 1.1.0
>
>
> If a message with a null key is produced to a compacted topic, the broker 
> returns `CorruptRecordException`, which is a retriable exception. As such, 
> the producer keeps retrying until retries are exhausted or request.timeout.ms 
> expires and eventually throws a TimeoutException. This is confusing and not 
> user-friendly.
> We should throw a meaningful error back to the user. From an implementation 
> perspective, we would have to use a non retriable error code to avoid this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3300) Calculate the initial/max size of offset index files and reduce the memory footprint for memory mapped index files.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3300:
---
Fix Version/s: (was: 1.0.0)

> Calculate the initial/max size of offset index files and reduce the memory 
> footprint for memory mapped index files.
> ---
>
> Key: KAFKA-3300
> URL: https://issues.apache.org/jira/browse/KAFKA-3300
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.1
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>
> Currently the initial/max size of offset index file is configured by 
> {{log.index.max.bytes}}. This will be the offset index file size for active 
> log segment until it rolls out. 
> Theoretically, we can calculate the upper bound of offset index size using 
> the following formula:
> {noformat}
> log.segment.bytes / index.interval.bytes * 8
> {noformat}
> With default setting the bytes needed for an offset index size is 1GB / 4K * 
> 8 = 2MB. And the default log.index.max.bytes is 10MB.
> This means we are over-allocating at least 8MB on disk and mapping it to 
> memory.
> We can probably do the following:
> 1. When creating a new offset index, calculate the size using the above 
> formula,
> 2. If the result in (1) is greater than log.index.max.bytes, we allocate 
> log.index.max.bytes instead.
> This should be able to significantly save memory if a broker has a lot of 
> partitions on it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3252) compression type for a topic should be used during log compaction

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3252:
---
Fix Version/s: (was: 1.0.0)

> compression type for a topic should be used during log compaction 
> --
>
> Key: KAFKA-3252
> URL: https://issues.apache.org/jira/browse/KAFKA-3252
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Ashish Singh
>
> Currently, the broker uses the specified compression type in a topic for 
> newly published messages. However, during log compaction, it still uses the 
> compression codec in the original message. To be consistent, it seems that we 
> should use the compression type in a topic when copying the messages to new 
> log segments.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3331) Refactor TopicCommand to make it testable and add unit tests

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3331:
---
Fix Version/s: (was: 1.0.0)

> Refactor TopicCommand to make it testable and add unit tests
> 
>
> Key: KAFKA-3331
> URL: https://issues.apache.org/jira/browse/KAFKA-3331
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.9.0.1
>Reporter: Ashish Singh
>Assignee: Ashish Singh
>
> TopicCommand has become a functionality packed, hard to read, class. Adding 
> or changing it with confidence requires some unit tests around it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3247) kafka-topic tool can use an option to see just "unbalanced partitions"

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3247:
---
Fix Version/s: (was: 1.0.0)

> kafka-topic tool can use an option to see just "unbalanced partitions"
> --
>
> Key: KAFKA-3247
> URL: https://issues.apache.org/jira/browse/KAFKA-3247
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>Assignee: Ashish Singh
>
> We already have an option for under replicated partitions, it will be nice to 
> also have the ability to just see partitions where the leader is not on the 
> preferred replica.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5986) Streams State Restoration never completes when logging is disabled

[Damian Guy (JIRA)]

Damian Guy created KAFKA-5986:
-

 Summary: Streams State Restoration never completes when logging is 
disabled
 Key: KAFKA-5986
 URL: https://issues.apache.org/jira/browse/KAFKA-5986
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.1
Reporter: Damian Guy
Assignee: Damian Guy
Priority: Critical
 Fix For: 1.0.0, 0.11.0.2


When logging is disabled on a state store, the store restoration never 
completes. This is likely because there are no changelogs, but more 
investigation is required.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-2651) Remove deprecated config alteration from TopicCommand in 0.9.1.0

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2651:
---
Fix Version/s: (was: 1.0.0)
   2.0.0

> Remove deprecated config alteration from TopicCommand in 0.9.1.0
> 
>
> Key: KAFKA-2651
> URL: https://issues.apache.org/jira/browse/KAFKA-2651
> Project: Kafka
>  Issue Type: Task
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Manikumar
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5759) Allow user to specify relative path as log directory

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5759:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Allow user to specify relative path as log directory
> 
>
> Key: KAFKA-5759
> URL: https://issues.apache.org/jira/browse/KAFKA-5759
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Critical
> Fix For: 1.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5018) LogCleaner tests to verify behaviour of message format v2

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5018:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> LogCleaner tests to verify behaviour of message format v2
> -
>
> Key: KAFKA-5018
> URL: https://issues.apache.org/jira/browse/KAFKA-5018
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
> Fix For: 1.1.0
>
>
> It would be good to add LogCleaner tests to verify the behaviour of fields 
> like baseOffset after compaction.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3190) KafkaProducer should not invoke callback in send()

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3190:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> KafkaProducer should not invoke callback in send()
> --
>
> Key: KAFKA-3190
> URL: https://issues.apache.org/jira/browse/KAFKA-3190
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Critical
> Fix For: 1.1.0
>
>
> Currently KafkaProducer will invoke callback.onComplete() if it receives an 
> ApiException during send(). This breaks the guarantee that callback will be 
> invoked in order. It seems ApiException in send() only comes from metadata 
> refresh. If so, we can probably simply throw it instead of invoking 
> callback().



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5347) OutOfSequence error should be fatal

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184012#comment-16184012
 ] 

Ismael Juma commented on KAFKA-5347:


[~apurva], is this still an issue?

> OutOfSequence error should be fatal
> ---
>
> Key: KAFKA-5347
> URL: https://issues.apache.org/jira/browse/KAFKA-5347
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
>  Labels: exactly-once
> Fix For: 1.1.0
>
>
> If the producer sees an OutOfSequence error for a given partition, we 
> currently treat it as an abortable error. This makes some sense because 
> OutOfSequence won't prevent us from being able to send the EndTxn to abort 
> the transaction. The problem is that the producer, even after aborting, still 
> won't be able to send to the topic with an OutOfSequence. One way to deal 
> with this is to ask the user to call {{initTransactions()}} again to bump the 
> epoch, but this is a bit difficult to explain and could be dangerous since it 
> renders zombie checking less effective. Probably we should just consider 
> OutOfSequence fatal for the transactional producer.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-3986) completedReceives can contain closed channels

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3986.

   Resolution: Fixed
Fix Version/s: (was: 1.1.0)
   1.0.0

> completedReceives can contain closed channels 
> --
>
> Key: KAFKA-3986
> URL: https://issues.apache.org/jira/browse/KAFKA-3986
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Reporter: Ryan P
> Fix For: 1.0.0
>
>
> I'm not entirely sure why at this point but it is possible to throw a Null 
> Pointer Exception when processingCompletedReceives. This happens when a 
> fairly substantial number of simultaneously initiated connections are 
> initiated with the server. 
> The processor thread does carry on but it may be worth investigating how the 
> channel could be both closed and  completedReceives. 
> The NPE in question is thrown here:
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala#L490
> It can not be consistently reproduced either. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-3986) completedReceives can contain closed channels

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184010#comment-16184010
 ] 

Ismael Juma commented on KAFKA-3986:


There have been many fixes to the Selector and SocketServer, so I am going to 
close this as "Fixed". If we see this again, let's reopen.

> completedReceives can contain closed channels 
> --
>
> Key: KAFKA-3986
> URL: https://issues.apache.org/jira/browse/KAFKA-3986
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Reporter: Ryan P
> Fix For: 1.0.0
>
>
> I'm not entirely sure why at this point but it is possible to throw a Null 
> Pointer Exception when processingCompletedReceives. This happens when a 
> fairly substantial number of simultaneously initiated connections are 
> initiated with the server. 
> The processor thread does carry on but it may be worth investigating how the 
> channel could be both closed and  completedReceives. 
> The NPE in question is thrown here:
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala#L490
> It can not be consistently reproduced either. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5879) Controller should read the latest IsrChangeNotification znodes when handling IsrChangeNotification event

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5879:
---
Fix Version/s: (was: 1.1.0)
   1.0.0

> Controller should read the latest IsrChangeNotification znodes when handling 
> IsrChangeNotification event
> 
>
> Key: KAFKA-5879
> URL: https://issues.apache.org/jira/browse/KAFKA-5879
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Critical
> Fix For: 1.0.0
>
>
> Currently controller can be very inefficient in handling 
> IsrChangeNotification event because it may need to access znode O(n^2) times 
> to handle O( n ) IsrChangeNotification znodes.
> For example, say there are 100 IsrChangeNotification nodes added to the 
> zookeeper. This will generate 100 IsrChangeNotification events with children 
> [1], [1, 2], [1, 2, 3], ... [1, 2, .. 100]. Let's say the controller now 
> needs to handle the IsrChangeNotification event with children [1, 2, ... 
> 100]. Controller will read zookeeper 100 times, delete these 100 znodes, 
> which further generates 100 IsrChangeNotification events with children count 
> from 0 to 99.
> The main cause of the problem is that, controller will attempt to access 
> zookeeper n times, where n is the count of the children at the time the 
> IsrChangeNotification event is generated, even though there is no 
> IsrChangeNotification znodes in the zookeeper.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5870) Idempotent producer: a producerId reset causes undesirable behavior for inflight batches to other partitions

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5870:
---
Labels: exactly-once  (was: )

> Idempotent producer: a producerId reset causes undesirable behavior for 
> inflight batches to other partitions
> 
>
> Key: KAFKA-5870
> URL: https://issues.apache.org/jira/browse/KAFKA-5870
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>  Labels: exactly-once
> Fix For: 1.1.0
>
>
> Currently, if we have to reset the producer id for any reason (for instance 
> if batches to a partition get expired, if we get an 
> {{OutOfOrderSequenceException}}, etc) we could cause batches to other 
> --healthy-- partitions to fail with a spurious 
> {{OutOfOrderSequenceException}}.
> This is detailed in this PR discussion: 
> https://github.com/apache/kafka/pull/3743#discussion_r137907630
> Ideally, we would want all inflight batches to be handled to completion 
> rather than potentially failing them prematurely. Further, since we want to 
> tighten up the semantics of the {{OutOfOrderSequenceException}}, at the very 
> least we should raise another exception in this case, because there is no 
> data loss on the broker when the client gives up. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5870) Idempotent producer: a producerId reset causes undesirable behavior for inflight batches to other partitions

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183998#comment-16183998
 ] 

Ismael Juma commented on KAFKA-5870:


[~apurva], just checking that it's OK to move this to 1.1.0?

> Idempotent producer: a producerId reset causes undesirable behavior for 
> inflight batches to other partitions
> 
>
> Key: KAFKA-5870
> URL: https://issues.apache.org/jira/browse/KAFKA-5870
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>  Labels: exactly-once
> Fix For: 1.1.0
>
>
> Currently, if we have to reset the producer id for any reason (for instance 
> if batches to a partition get expired, if we get an 
> {{OutOfOrderSequenceException}}, etc) we could cause batches to other 
> --healthy-- partitions to fail with a spurious 
> {{OutOfOrderSequenceException}}.
> This is detailed in this PR discussion: 
> https://github.com/apache/kafka/pull/3743#discussion_r137907630
> Ideally, we would want all inflight batches to be handled to completion 
> rather than potentially failing them prematurely. Further, since we want to 
> tighten up the semantics of the {{OutOfOrderSequenceException}}, at the very 
> least we should raise another exception in this case, because there is no 
> data loss on the broker when the client gives up. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5843) Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to true

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5843:
---
Fix Version/s: (was: 1.1.0)
   1.0.0

> Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to 
> true
> ---
>
> Key: KAFKA-5843
> URL: https://issues.apache.org/jira/browse/KAFKA-5843
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5829) Speedup broker startup after unclean shutdown by reducing unnecessary snapshot files deletion

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5829:
---
Fix Version/s: (was: 1.1.0)
   1.0.0

> Speedup broker startup after unclean shutdown by reducing unnecessary 
> snapshot files deletion
> -
>
> Key: KAFKA-5829
> URL: https://issues.apache.org/jira/browse/KAFKA-5829
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Critical
> Fix For: 1.0.0
>
>
> The current Kafka implementation will cause slow startup after unclean 
> shutdown. The time to load a partition will be 10X or more than what it 
> actually needs. Here is the explanation with example:
> - Say we have a partition of 20 segments, each segment has 250 message 
> starting with offset 0. And each message has 1 MB bytes.
> - Broker experiences hard kill and the index file of the first segment is 
> corrupted.
> - When broker startup and load the first segment, it realizes that the index 
> of the first segment is corrupted. So it calls `log.recoverSegment(...)` to 
> recover this segment. This method will call 
> `stateManager.truncateAndReload(...)` which deletes the snapshot files whose 
> offset is larger than base offset of the first segment. Thus all snapshot 
> files are deleted.
> - To rebuild the snapshot files, the `log.loadSegmentFiles(...)` will have to 
> read every message in this partition even if their log and index files are 
> not corrupted. This will increase the time to load this partition by more 
> than an order of magnitude.
> In order to address this issue, one simple solution is not to delete snapshot 
> files that are than the given offset if only the index files needs re-build. 
> More specifically, we should not need to re-build producer state offset file 
> unless the log file itself is corrupted or truncated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5503) Idempotent producer ignores shutdown while fetching ProducerId

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5503:
---
Fix Version/s: 1.0.1

> Idempotent producer ignores shutdown while fetching ProducerId
> --
>
> Key: KAFKA-5503
> URL: https://issues.apache.org/jira/browse/KAFKA-5503
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Affects Versions: 0.11.0.0
>Reporter: Jason Gustafson
>Assignee: Evgeny Veretennikov
> Fix For: 1.1.0, 1.0.1
>
>
> When using the idempotent producer, we initially block the sender thread 
> while we attempt to get the ProducerId. During this time, a concurrent call 
> to close() will be ignored.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5212) Consumer ListOffsets request can starve group heartbeats

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5212:
---
Fix Version/s: 1.0.1

> Consumer ListOffsets request can starve group heartbeats
> 
>
> Key: KAFKA-5212
> URL: https://issues.apache.org/jira/browse/KAFKA-5212
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Jason Gustafson
> Fix For: 1.1.0, 1.0.1
>
>
> The consumer is not able to send heartbeats while it is awaiting a 
> ListOffsets response. Typically this is not a problem because ListOffsets 
> requests are handled quickly, but in the worst case if the request takes 
> longer than the session timeout, the consumer will fall out of the group.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5153) KAFKA Cluster : 0.10.2.0 : Servers Getting disconnected : Service Impacting

[Sandeep Kale (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183993#comment-16183993
 ] 

Sandeep Kale commented on KAFKA-5153:
-

[~arpan.khagram0...@gmail.com] Any update on this? How u managed to get 
resolution for this? 

> KAFKA Cluster : 0.10.2.0 : Servers Getting disconnected : Service Impacting
> ---
>
> Key: KAFKA-5153
> URL: https://issues.apache.org/jira/browse/KAFKA-5153
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: RHEL 6
> Java Version  1.8.0_91-b14
>Reporter: Arpan
>Priority: Critical
> Attachments: server_1_72server.log, server_2_73_server.log, 
> server_3_74Server.log, server.properties, ThreadDump_1493564142.dump, 
> ThreadDump_1493564177.dump, ThreadDump_1493564249.dump
>
>
> Hi Team, 
> I was earlier referring to issue KAFKA-4477 because the problem i am facing 
> is similar. I tried to search the same reference in release docs as well but 
> did not get anything in 0.10.1.1 or 0.10.2.0. I am currently using 
> 2.11_0.10.2.0.
> I am have 3 node cluster for KAFKA and cluster for ZK as well on the same set 
> of servers in cluster mode. We are having around 240GB of data getting 
> transferred through KAFKA everyday. What we are observing is disconnect of 
> the server from cluster and ISR getting reduced and it starts impacting 
> service.
> I have also observed file descriptor count getting increased a bit, in normal 
> circumstances we have not observed FD count more than 500 but when issue 
> started we were observing it in the range of 650-700 on all 3 servers. 
> Attaching thread dumps of all 3 servers when we started facing the issue 
> recently.
> The issue get vanished once you bounce the nodes and the set up is not 
> working more than 5 days without this issue. Attaching server logs as well.
> Kindly let me know if you need any additional information. Attaching 
> server.properties as well for one of the server (It's similar on all 3 
> serversP)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-4423) Drop support for Java 7

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4423:
---
Fix Version/s: (was: 1.1.0)
   2.0.0

> Drop support for Java 7
> ---
>
> Key: KAFKA-4423
> URL: https://issues.apache.org/jira/browse/KAFKA-4423
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>  Labels: kip
> Fix For: 2.0.0
>
>
> Java 7 was released in July 2011, it hasn't received public updates since 
> April 2015, Java 8 was released in March 2014 and Java 9 is scheduled to be 
> released in July 2017.
> The last public release of JDK 7 by Oracle contains a large number of known 
> security vulnerabilities and Java 8 introduces a number of
> compelling features and we will soon have to support Java 9 so it would be 
> good to drop support for Java 7 in 2017. The actual timing would depend on 
> when we release the next major release of Kafka.
> More details can be found in the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-118%3A+Drop+Support+for+Java+7+in+Kafka+0.11



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5864) ReplicaFetcherThread should not die due to replica in offline log directory

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5864:
---
Fix Version/s: (was: 1.1.0)
   1.0.0

> ReplicaFetcherThread should not die due to replica in offline log directory
> ---
>
> Key: KAFKA-5864
> URL: https://issues.apache.org/jira/browse/KAFKA-5864
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5552) testTransactionalProducerTopicAuthorizationExceptionInCommit fails

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183990#comment-16183990
 ] 

Ismael Juma commented on KAFKA-5552:


Not sure if this is still possible, cc [~apurva] to check.

> testTransactionalProducerTopicAuthorizationExceptionInCommit fails 
> ---
>
> Key: KAFKA-5552
> URL: https://issues.apache.org/jira/browse/KAFKA-5552
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Eno Thereska
> Fix For: 1.1.0
>
>
> Got a unit test error: 
> https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5877/
> Error Message
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
> Stacktrace
> org.apache.kafka.common.KafkaException: Cannot execute transactional method 
> because we are in an error state
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:524)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.beginCommit(TransactionManager.java:190)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.commitTransaction(KafkaProducer.java:583)
>   at 
> kafka.api.AuthorizerIntegrationTest.testTransactionalProducerTopicAuthorizationExceptionInCommit(AuthorizerIntegrationTest.scala:1027)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
>   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> 

[jira] [Updated] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3177:
---
Fix Version/s: (was: 1.0.0)
   1.1.0

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 1.1.0
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2875:
---
Fix Version/s: (was: 1.0.0)

> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: jin xing
>Priority: Minor
>  Labels: patch
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5764) KafkaShortnamer should allow for case-insensitive matches

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183983#comment-16183983
 ] 

Ismael Juma commented on KAFKA-5764:


The KIP missed the freeze deadline, so updated the fix version.

> KafkaShortnamer should allow for case-insensitive matches 
> --
>
> Key: KAFKA-5764
> URL: https://issues.apache.org/jira/browse/KAFKA-5764
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.11.0.0
>Reporter: Ryan P
>Assignee: Manikumar
> Fix For: 1.1.0
>
>
> Currently it does not appear that the KafkaShortnamer allows for case 
> insensitive search and replace rules. 
> It would be good to match the functionality provided by HDFS as operators are 
> familiar with this. This also makes it easier to port auth_to_local rules 
> from your existing hdfs configurations to your new kafka configuration. 
> HWX auth_to_local guide for reference
> https://community.hortonworks.com/articles/14463/auth-to-local-rules-syntax.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5976) RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE logging

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5976.

   Resolution: Fixed
Fix Version/s: 1.0.0

> RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE 
> logging
> 
>
> Key: KAFKA-5976
> URL: https://issues.apache.org/jira/browse/KAFKA-5976
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.1
>Reporter: huxihx
>Assignee: huxihx
> Fix For: 1.0.0
>
>
> In RequestChannel.scala, RequestChannel.sendResponse records incorrect size 
> for `NetworkSend` when trace logging is enabled, as shown below:
> {code:title=RequestChannel.scala|borderStyle=solid}
> def sendResponse(response: RequestChannel.Response) {
> if (isTraceEnabled) {
>   val requestHeader = response.request.header
>   trace(s"Sending ${requestHeader.apiKey} response to client 
> ${requestHeader.clientId} of " + s"${response.responseSend.size} bytes.")
> }
> {code}
> `responseSend` is of `scala.Option` type so it should be 
> `response.responseSend.get.size`. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5976) RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE logging

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183964#comment-16183964
 ] 

ASF GitHub Bot commented on KAFKA-5976:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3961


> RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE 
> logging
> 
>
> Key: KAFKA-5976
> URL: https://issues.apache.org/jira/browse/KAFKA-5976
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.1
>Reporter: huxihx
>Assignee: huxihx
>
> In RequestChannel.scala, RequestChannel.sendResponse records incorrect size 
> for `NetworkSend` when trace logging is enabled, as shown below:
> {code:title=RequestChannel.scala|borderStyle=solid}
> def sendResponse(response: RequestChannel.Response) {
> if (isTraceEnabled) {
>   val requestHeader = response.request.header
>   trace(s"Sending ${requestHeader.apiKey} response to client 
> ${requestHeader.clientId} of " + s"${response.responseSend.size} bytes.")
> }
> {code}
> `responseSend` is of `scala.Option` type so it should be 
> `response.responseSend.get.size`. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5979) kip-120 changed internal streams topic naming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183951#comment-16183951
 ] 

ASF GitHub Bot commented on KAFKA-5979:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3979


> kip-120 changed internal streams topic naming
> -
>
> Key: KAFKA-5979
> URL: https://issues.apache.org/jira/browse/KAFKA-5979
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: dan norwood
>Assignee: Matthias J. Sax
>Priority: Blocker
> Fix For: 1.0.0
>
>
> prior to kip-120 i had a join that made a topic named 
> `KSTREAM-OUTEROTHER-96-store`, after 120 it is named 
> `KSTREAM-OUTEROTHER-84-store`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4914) Partition re-assignment tool should check types before persisting state in ZooKeeper

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183948#comment-16183948
 ] 

Ismael Juma commented on KAFKA-4914:


I am hoping to get this in before the code freeze since it's a validation bug 
fix (i.e. low risk). Sorry for the delay.

> Partition re-assignment tool should check types before persisting state in 
> ZooKeeper
> 
>
> Key: KAFKA-4914
> URL: https://issues.apache.org/jira/browse/KAFKA-4914
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.10.1.1
>Reporter: Nick Travers
>Assignee: Nick Travers
> Fix For: 1.0.0
>
>
> The partition-reassignment too currently allows non-type-safe information to 
> be persisted into ZooKeeper, which can result in a ClassCastException at 
> runtime for brokers.
> Specifically, this occurred when the broker assignment field was a List of 
> Strings, instead of a List of Integers.
> {code}
> 2017-03-15 01:44:04,572 ERROR 
> [ZkClient-EventThread-36-samsa-zkserver.stage.sjc1.square:26101/samsa] 
> controller.ReplicaStateMachine$BrokerChangeListener - [BrokerChangeListener 
> on Controller 10]: Error while handling broker changes
> java.lang.ClassCastException: java.lang.String cannot be cast to 
> java.lang.Integer
> at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101)
> at 
> kafka.controller.KafkaController$$anonfun$8$$anonfun$apply$2.apply(KafkaController.scala:436)
> at 
> scala.collection.LinearSeqOptimized$class.exists(LinearSeqOptimized.scala:93)
> at scala.collection.immutable.List.exists(List.scala:84)
> at 
> kafka.controller.KafkaController$$anonfun$8.apply(KafkaController.scala:436)
> at 
> kafka.controller.KafkaController$$anonfun$8.apply(KafkaController.scala:435)
> at 
> scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at 
> scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
> at 
> scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
> at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
> at 
> kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:435)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:374)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:358)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:358)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:357)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:356)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:356)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:355)
> at org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:843)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (KAFKA-4914) Partition re-assignment tool should check types before persisting state in ZooKeeper

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-4914:
--

Assignee: Nick Travers

> Partition re-assignment tool should check types before persisting state in 
> ZooKeeper
> 
>
> Key: KAFKA-4914
> URL: https://issues.apache.org/jira/browse/KAFKA-4914
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.10.1.1
>Reporter: Nick Travers
>Assignee: Nick Travers
> Fix For: 1.0.0
>
>
> The partition-reassignment too currently allows non-type-safe information to 
> be persisted into ZooKeeper, which can result in a ClassCastException at 
> runtime for brokers.
> Specifically, this occurred when the broker assignment field was a List of 
> Strings, instead of a List of Integers.
> {code}
> 2017-03-15 01:44:04,572 ERROR 
> [ZkClient-EventThread-36-samsa-zkserver.stage.sjc1.square:26101/samsa] 
> controller.ReplicaStateMachine$BrokerChangeListener - [BrokerChangeListener 
> on Controller 10]: Error while handling broker changes
> java.lang.ClassCastException: java.lang.String cannot be cast to 
> java.lang.Integer
> at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101)
> at 
> kafka.controller.KafkaController$$anonfun$8$$anonfun$apply$2.apply(KafkaController.scala:436)
> at 
> scala.collection.LinearSeqOptimized$class.exists(LinearSeqOptimized.scala:93)
> at scala.collection.immutable.List.exists(List.scala:84)
> at 
> kafka.controller.KafkaController$$anonfun$8.apply(KafkaController.scala:436)
> at 
> kafka.controller.KafkaController$$anonfun$8.apply(KafkaController.scala:435)
> at 
> scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at 
> scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
> at 
> scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
> at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
> at 
> kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:435)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:374)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:358)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:358)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:357)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:356)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:356)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:355)
> at org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:843)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5949) User Callback Exceptions need to be handled properly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183947#comment-16183947
 ] 

ASF GitHub Bot commented on KAFKA-5949:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3939


> User Callback Exceptions need to be handled properly
> 
>
> Key: KAFKA-5949
> URL: https://issues.apache.org/jira/browse/KAFKA-5949
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 1.0.0
>
>
> In Streams, we allow to register multiple user callbacks. We need to handle 
> those exceptions gracefully, by catching and wrapping with a StreamsException.
> - TimestampExtractor
> - DeserializationHandler
> - StateRestoreListener



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-3186) KIP-50: Move Authorizer and related classes to separate package.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183945#comment-16183945
 ] 

Ismael Juma commented on KAFKA-3186:


KIP-50 was never merged [~guozhang].

> KIP-50: Move Authorizer and related classes to separate package.
> 
>
> Key: KAFKA-3186
> URL: https://issues.apache.org/jira/browse/KAFKA-3186
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Ashish Singh
>Assignee: Ashish Singh
>
> [KIP-50|https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Move+Authorizer+to+a+separate+package]
>  has more details.
> Kafka supports pluggable authorization. Third party authorizer 
> implementations allow existing authorization systems like, Apache Sentry, 
> Apache Ranger, etc to extend authorization to Kafka as well. Implementing 
> Kafka's authorizer interface requires depending on kafka's core, which is 
> huge. This has been already raised as a concern by Sentry, Ranger and Kafka 
> community. Even Kafka clients require duplication of authorization related 
> classes, like Resource, Operation, etc, for adding ACLs CRUD APIs.
> Kafka authorizer is agnostic of principal types it supports, so are the acls 
> CRUD methods in Authorizer interface. The intent behind is to keep Kafka 
> principal types pluggable, which is really great. However, this leads to Acls 
> CRUD methods not performing any check on validity of acls, as they are not 
> aware of what principal types Authorizer implementation supports. This opens 
> up space for lots of user errors, KAFKA-3097 is an instance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-3186) KIP-50: Move Authorizer and related classes to separate package.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3186:
---
Fix Version/s: (was: 1.0.0)

> KIP-50: Move Authorizer and related classes to separate package.
> 
>
> Key: KAFKA-3186
> URL: https://issues.apache.org/jira/browse/KAFKA-3186
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Ashish Singh
>Assignee: Ashish Singh
>
> [KIP-50|https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Move+Authorizer+to+a+separate+package]
>  has more details.
> Kafka supports pluggable authorization. Third party authorizer 
> implementations allow existing authorization systems like, Apache Sentry, 
> Apache Ranger, etc to extend authorization to Kafka as well. Implementing 
> Kafka's authorizer interface requires depending on kafka's core, which is 
> huge. This has been already raised as a concern by Sentry, Ranger and Kafka 
> community. Even Kafka clients require duplication of authorization related 
> classes, like Resource, Operation, etc, for adding ACLs CRUD APIs.
> Kafka authorizer is agnostic of principal types it supports, so are the acls 
> CRUD methods in Authorizer interface. The intent behind is to keep Kafka 
> principal types pluggable, which is really great. However, this leads to Acls 
> CRUD methods not performing any check on validity of acls, as they are not 
> aware of what principal types Authorizer implementation supports. This opens 
> up space for lots of user errors, KAFKA-3097 is an instance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)