[jira] [Commented] (KAFKA-5593) Kafka streams not re-balancing when 3 consumer streams are there

[Yogesh BG (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091135#comment-16091135
 ] 

Yogesh BG commented on KAFKA-5593:
--

Hey thanks fro pointing down that. Yes you are correct. I changed the 
configuration and verified it works.

> Kafka streams not re-balancing when 3 consumer streams are there
> 
>
> Key: KAFKA-5593
> URL: https://issues.apache.org/jira/browse/KAFKA-5593
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Yogesh BG
>Priority: Critical
> Attachments: log1.txt, log2.txt, log3.txt
>
>
> I have 3 broker nodes, 3 kafka streams
> I observe that all 3 consumer streams are part of the group named 
> rtp-kafkastreams. but when i see the data is processed only by one node. 
> 
> DEBUG n.a.a.k.a.AccessLogMetricEnrichmentProcessor - 
> AccessLogMetricEnrichmentProcessor.process
> when i do check the partition information shared by each of them i see first 
> node has all partitions like all 8. but in other streams the folder is empty.
> 
> [root@ip-172-31-11-139 ~]# ls /data/kstreams/rtp-kafkastreams
> 0_0  0_1  0_2  0_3  0_4  0_5  0_6  0_7
>  and  this folder is empty
> I tried restarting the other two consumer streams still they won't become the 
> part of the group and re-balance.
> I have attached the logs.
> Configurations are inside the log file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-2527) System Test for Quotas in Ducktape

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091085#comment-16091085
 ] 

ASF GitHub Bot commented on KAFKA-2527:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3447


> System Test for Quotas in Ducktape
> --
>
> Key: KAFKA-2527
> URL: https://issues.apache.org/jira/browse/KAFKA-2527
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dong Lin
>Assignee: Dong Lin
>  Labels: quota
> Fix For: 0.9.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5605) Multiple KStreams being created on application startup are not processing

[Connie Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Connie Chen resolved KAFKA-5605.

Resolution: Not A Problem

> Multiple KStreams being created on application startup are not processing
> -
>
> Key: KAFKA-5605
> URL: https://issues.apache.org/jira/browse/KAFKA-5605
> Project: Kafka
>  Issue Type: Bug
>Reporter: Connie Chen
> Attachments: app-logs.txt, server-logs.txt
>
>
> This is an issue filed in response to this slack discussion
> https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627
> I have a http service on top of Akka that starts and stops KStreams whose 
> source/sink topics are configurable. Every time a new KStream is created I 
> post a record to another Kafka topic `StreamConfig2`. On my application 
> startup, I read from that config topic into a KTable and iterate records, 
> create one KStream per record. However, it appears that this startup creation 
> is not deterministic (ie. sometimes the KStreams get created properly, 
> sometimes not). Also, not all KStreams are created. 
> Matthias helpfully suggested that I change `application.id` every time on my 
> application startup as a workaround. This seems to work for my application 
> (service startup always creates the existing KStream in config topic), 
> however I can't figure out why it doesn't work when I reuse `application.id`
> I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried 
> increasing the config there, to no avail. 
> Here I've attached logs from my application, expected logs, and kafka server 
> logs. 
> https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5605) Multiple KStreams being created on application startup are not processing

[Connie Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090812#comment-16090812
 ] 

Connie Chen commented on KAFKA-5605:


[~mjsax] helped me fix the issue. It appears I was sharing `application.id` 
across all my `KafkaStream` instances in my application. Each instance of 
`KafkaStream` under the same `application.id` must be processing the same 
topics. So the fix is to create a unique `application.id` per topic. Then, my 
service application will have N application.id processing N different topics, 
each with a different transformation, and each with its own `KStreamBuilder` 
and `KafkaStream`

https://github.com/conniec/examples/commit/ede9051734f414ce3bae5d760789d3fc9f60e769

> Multiple KStreams being created on application startup are not processing
> -
>
> Key: KAFKA-5605
> URL: https://issues.apache.org/jira/browse/KAFKA-5605
> Project: Kafka
>  Issue Type: Bug
>Reporter: Connie Chen
> Attachments: app-logs.txt, server-logs.txt
>
>
> This is an issue filed in response to this slack discussion
> https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627
> I have a http service on top of Akka that starts and stops KStreams whose 
> source/sink topics are configurable. Every time a new KStream is created I 
> post a record to another Kafka topic `StreamConfig2`. On my application 
> startup, I read from that config topic into a KTable and iterate records, 
> create one KStream per record. However, it appears that this startup creation 
> is not deterministic (ie. sometimes the KStreams get created properly, 
> sometimes not). Also, not all KStreams are created. 
> Matthias helpfully suggested that I change `application.id` every time on my 
> application startup as a workaround. This seems to work for my application 
> (service startup always creates the existing KStream in config topic), 
> however I can't figure out why it doesn't work when I reuse `application.id`
> I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried 
> increasing the config there, to no avail. 
> Here I've attached logs from my application, expected logs, and kafka server 
> logs. 
> https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5587) Processor got uncaught exception: NullPointerException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090766#comment-16090766
 ] 

ASF GitHub Bot commented on KAFKA-5587:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3526


> Processor got uncaught exception: NullPointerException
> --
>
> Key: KAFKA-5587
> URL: https://issues.apache.org/jira/browse/KAFKA-5587
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.1
>Reporter: Dan
>Assignee: Rajini Sivaram
>
> [2017-07-12 21:56:39,964] ERROR Processor got uncaught exception. 
> (kafka.network.Processor)
> java.lang.NullPointerException
> at 
> kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:490)
> at 
> kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:487)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at 
> kafka.network.Processor.processCompletedReceives(SocketServer.scala:487)
> at kafka.network.Processor.run(SocketServer.scala:417)
> at java.lang.Thread.run(Thread.java:745)
> Anyone knows the cause of this exception? What's the effect of it? 
> When this exception occurred, the log also showed that the broker was 
> frequently shrinking ISR to itself. Are these two things interrelated?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5605) Multiple KStreams being created on application startup are not processing

[Connie Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Connie Chen updated KAFKA-5605:
---
Description: 
This is an issue filed in response to this slack discussion
https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627

I have a http service on top of Akka that starts and stops KStreams whose 
source/sink topics are configurable. Every time a new KStream is created I post 
a record to another Kafka topic `StreamConfig2`. On my application startup, I 
read from that config topic into a KTable and iterate records, create one 
KStream per record. However, it appears that this startup creation is not 
deterministic (ie. sometimes the KStreams get created properly, sometimes not). 
Also, not all KStreams are created. 

Matthias helpfully suggested that I change `application.id` every time on my 
application startup as a workaround. This seems to work for my application 
(service startup always creates the existing KStream in config topic), however 
I can't figure out why it doesn't work when I reuse `application.id`

I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried 
increasing the config there, to no avail. 

Here I've attached logs from my application, expected logs, and kafka server 
logs. 
https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e

  was:
This is an issue filed in response to this slack discussion
https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627

I have a http service on top of Akka that starts and stops KStreams whose 
source/sink topics are configurable. Every time a new KStream is created I post 
a record to another Kafka topic `StreamConfig2`. On my application startup, I 
read from that config topic into a KTable and create existing KStreams. 
However, it appears that this startup creation is not deterministic (ie. 
sometimes the KStreams get created properly, sometimes not). 

Matthias helpfully suggested that I change `application.id` every time on my 
application startup as a workaround. This seems to work for my application 
(service startup always creates the existing KStream in config topic), however 
I can't figure out why it doesn't work when I reuse `application.id`

I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried 
increasing the config there, to no avail. 

Here I've attached logs from my application, expected logs, and kafka server 
logs. 
https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e


> Multiple KStreams being created on application startup are not processing
> -
>
> Key: KAFKA-5605
> URL: https://issues.apache.org/jira/browse/KAFKA-5605
> Project: Kafka
>  Issue Type: Bug
>Reporter: Connie Chen
> Attachments: app-logs.txt, server-logs.txt
>
>
> This is an issue filed in response to this slack discussion
> https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627
> I have a http service on top of Akka that starts and stops KStreams whose 
> source/sink topics are configurable. Every time a new KStream is created I 
> post a record to another Kafka topic `StreamConfig2`. On my application 
> startup, I read from that config topic into a KTable and iterate records, 
> create one KStream per record. However, it appears that this startup creation 
> is not deterministic (ie. sometimes the KStreams get created properly, 
> sometimes not). Also, not all KStreams are created. 
> Matthias helpfully suggested that I change `application.id` every time on my 
> application startup as a workaround. This seems to work for my application 
> (service startup always creates the existing KStream in config topic), 
> however I can't figure out why it doesn't work when I reuse `application.id`
> I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried 
> increasing the config there, to no avail. 
> Here I've attached logs from my application, expected logs, and kafka server 
> logs. 
> https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5605) Multiple KStreams being created on application startup are not processing

[Connie Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Connie Chen updated KAFKA-5605:
---
Summary: Multiple KStreams being created on application startup are not 
processing  (was: KStream being created on application startup is not 
processing)

> Multiple KStreams being created on application startup are not processing
> -
>
> Key: KAFKA-5605
> URL: https://issues.apache.org/jira/browse/KAFKA-5605
> Project: Kafka
>  Issue Type: Bug
>Reporter: Connie Chen
> Attachments: app-logs.txt, server-logs.txt
>
>
> This is an issue filed in response to this slack discussion
> https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627
> I have a http service on top of Akka that starts and stops KStreams whose 
> source/sink topics are configurable. Every time a new KStream is created I 
> post a record to another Kafka topic `StreamConfig2`. On my application 
> startup, I read from that config topic into a KTable and create existing 
> KStreams. However, it appears that this startup creation is not deterministic 
> (ie. sometimes the KStreams get created properly, sometimes not). 
> Matthias helpfully suggested that I change `application.id` every time on my 
> application startup as a workaround. This seems to work for my application 
> (service startup always creates the existing KStreams), however I can't 
> figure out why it doesn't work when I reuse `application.id`
> I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried 
> increasing the config there, to no avail. 
> Here I've attached logs from my application, expected logs, and kafka server 
> logs. 
> https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5605) KStream being created on application startup is not processing

[Connie Chen (JIRA)]

Connie Chen created KAFKA-5605:
--

 Summary: KStream being created on application startup is not 
processing
 Key: KAFKA-5605
 URL: https://issues.apache.org/jira/browse/KAFKA-5605
 Project: Kafka
  Issue Type: Bug
Reporter: Connie Chen
 Attachments: app-logs.txt, server-logs.txt

This is an issue filed in response to this slack discussion
https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627

I have a http service on top of Akka that starts and stops KStreams whose 
source/sink topics are configurable. Every time a new KStream is created I post 
a record to another Kafka topic `StreamConfig2`. On my application startup, I 
read from that config topic into a KTable and create existing KStreams. 
However, it appears that this startup creation is not deterministic (ie. 
sometimes the KStreams get created properly, sometimes not). 

Matthias helpfully suggested that I change `application.id` every time on my 
application startup as a workaround. This seems to work for my application 
(service startup always creates the existing KStreams), however I can't figure 
out why it doesn't work when I reuse `application.id`

I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried 
increasing the config there, to no avail. 

Here I've attached logs from my application, expected logs, and kafka server 
logs. 
https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5604) All producer methods should raise `ProducerFencedException` after the first time.

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5604:
---

 Summary: All producer methods should raise 
`ProducerFencedException` after the first time.
 Key: KAFKA-5604
 URL: https://issues.apache.org/jira/browse/KAFKA-5604
 Project: Kafka
  Issue Type: Bug
Reporter: Apurva Mehta
Assignee: Apurva Mehta


Currently, when a `ProducerFencedException` is raised from a transactional 
producer, the expectation is that the application should call `close` 
immediately. However, if the application calls other producer methods, they 
would get a `KafkaException`. This is a bit confusing, and results in tickets 
like : https://issues.apache.org/jira/browse/KAFKA-5603. 

We should update the producer so that calls to any method other than `close` 
should raise a `ProducerFencedException` after the first time it is raised.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5603) Streams should not abort transaction when closing zombie task

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-5603:
--

 Summary: Streams should not abort transaction when closing zombie 
task
 Key: KAFKA-5603
 URL: https://issues.apache.org/jira/browse/KAFKA-5603
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.0
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
Priority: Critical
 Fix For: 0.11.0.1


The contract of the transactional producer API is to not call any transactional 
method after a {{ProducerFenced}} exception was thrown.

Streams however, does an unconditional call within {{StreamTask#close()}} to 
{{abortTransaction()}} in case of unclean shutdown. We need to distinguish 
between a {{ProducerFenced}} and other unclean shutdown cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5600) GroupMetadataManager doesn't read offsets of segmented logs correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090330#comment-16090330
 ] 

ASF GitHub Bot commented on KAFKA-5600:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3538


> GroupMetadataManager doesn't read offsets of segmented logs correctly
> -
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
>Priority: Critical
>  Labels: regression, reliability
> Fix For: 0.11.0.1
>
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets, consumers will start at a wrong position when 
> metadata and offset events are in both segments.
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
> 5mins). The close of the consumer is needed to have metadata events in the 
> segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
> commit 
> https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32
> I will prepare a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5595) Illegal state in SocketServer; attempt to send with another send in progress

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090205#comment-16090205
 ] 

Jason Gustafson commented on KAFKA-5595:


Ensuring connectionId uniqueness is a nice improvement, but I agree with Ismael 
that it seems unlikely to fix the underlying issue. We may have to be content 
with the Rajini's logging improvements for now and hope that they will give us 
some more hints the next time it happens. Seems extremely rare in any case. 

One other data point worth mentioning: I mentioned the "Attempting to send 
response via channel ..." messages above. In fact, there were about 13K of 
these messages within 1 second of the illegal state exceptions. 

> Illegal state in SocketServer; attempt to send with another send in progress
> 
>
> Key: KAFKA-5595
> URL: https://issues.apache.org/jira/browse/KAFKA-5595
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>
> I have seen this a couple times, but I'm not sure the conditions associated 
> with it. 
> {code}
> java.lang.IllegalStateException: Attempt to begin a send operation with prior 
> send operation still in progress.
>   at 
> org.apache.kafka.common.network.KafkaChannel.setSend(KafkaChannel.java:138)
>   at org.apache.kafka.common.network.Selector.send(Selector.java:248)
>   at kafka.network.Processor.sendResponse(SocketServer.scala:488)
>   at kafka.network.Processor.processNewResponses(SocketServer.scala:466)
>   at kafka.network.Processor.run(SocketServer.scala:431)
>   at java.lang.Thread.run(Thread.java:748)
> {code}
> Prior to this event, I see a lot of this message in the logs (always for the 
> same connection id):
> {code}
> Attempting to send response via channel for which there is no open 
> connection, connection id 7
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5579) SchemaBuilder.type(Schema.Type) should not allow null.

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-5579.
--
   Resolution: Fixed
Fix Version/s: 0.11.1.0

Issue resolved by pull request 3517
[https://github.com/apache/kafka/pull/3517]

> SchemaBuilder.type(Schema.Type) should not allow null.
> --
>
> Key: KAFKA-5579
> URL: https://issues.apache.org/jira/browse/KAFKA-5579
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Jeremy Custenborder
>Assignee: Jeremy Custenborder
>Priority: Minor
> Fix For: 0.11.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5545) Kafka Stream not able to successfully restart over new broker ip

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090111#comment-16090111
 ] 

Guozhang Wang commented on KAFKA-5545:
--

Hello [~yogeshbelur] sorry for the late reply.

I think I have found the root cause of the issue, which is indeed a deadlock 
that cause the stream thread to be not able to exit its current loop call and 
hence not be able to shutdown. More details are the following:

1. When {{stream.close()}} is called, it does nothing but just set a flag in 
the stream threads and let the shutdown thread to wait until the stream threads 
have seen this flag and end the while loop and shutdown themselves.
2. There is an edge case, that the stream thread is still in the 
{{onPartitionAssigned}} callback of the current loop, and is trying to grab the 
state directory locks in order to create the assigned tasks; however the state 
directory locks are not released by other threads who have seen a disconnect / 
request timeout from the brokers since the broker IP has changed within the 
same JVM, they will retry indefinitely as the rebalance timeout is default to 
{{Integer.MAX_VALUE}}. So the thread will never complete its current iteration 
and will not see the shutdown flag and will not shutdown itself.

I have started proposing a fix for the state directory deadlock now, but this 
will probably only in 0.11.0.1 release and backported if we are having an 
0.10.2.2 release. As for the current version (0.10.2.1) version you are on. One 
walkaround is to set the {{ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG}} to a 
smaller value instead of the default {{Integer.MAX_VALUE}} so that it will not 
cause a dead lock, and making sure that for whatever this config value is set 
to, the corresponding {{REQUEST_TIMEOUT_MS_CONFIG}} is set to be larger than it.


> Kafka Stream not able to successfully restart over new broker ip
> 
>
> Key: KAFKA-5545
> URL: https://issues.apache.org/jira/browse/KAFKA-5545
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Yogesh BG
>Priority: Critical
> Attachments: kafkastreams.log
>
>
> Hi
> I have one kafka broker and one kafka stream application
> initially kafka stream connected and starts processing data. Then i restart 
> the broker. When broker restarts new ip will be assigned.
> In kafka stream i have a 5min interval thread which checks if broker ip 
> changed and if changed, we cleanup the stream, rebuild topology(tried with 
> reusing topology) and start the stream again. I end up with the following 
> exceptions.
> 11:04:08.032 [StreamThread-38] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-38] Creating active task 0_5 with assigned 
> partitions [PR-5]
> 11:04:08.033 [StreamThread-41] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-41] Creating active task 0_1 with assigned 
> partitions [PR-1]
> 11:04:08.036 [StreamThread-34] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-34] Creating active task 0_7 with assigned 
> partitions [PR-7]
> 11:04:08.036 [StreamThread-37] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-37] Creating active task 0_3 with assigned 
> partitions [PR-3]
> 11:04:08.036 [StreamThread-45] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-45] Creating active task 0_0 with assigned 
> partitions [PR-0]
> 11:04:08.037 [StreamThread-36] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-36] Creating active task 0_4 with assigned 
> partitions [PR-4]
> 11:04:08.037 [StreamThread-43] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-43] Creating active task 0_6 with assigned 
> partitions [PR-6]
> 11:04:08.038 [StreamThread-48] INFO  o.a.k.s.p.internals.StreamThread - 
> stream-thread [StreamThread-48] Creating active task 0_2 with assigned 
> partitions [PR-2]
> 11:04:09.034 [StreamThread-38] WARN  o.a.k.s.p.internals.StreamThread - Could 
> not create task 0_5. Will retry.
> org.apache.kafka.streams.errors.LockException: task [0_5] Failed to lock the 
> state directory for task 0_5
> at 
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.(ProcessorStateManager.java:100)
>  ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at 
> org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:73)
>  ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:108)
>  ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na]
> at 
> 

[jira] [Commented] (KAFKA-5602) Add --custom-ducktape flag to ducker-ak

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090094#comment-16090094
 ] 

ASF GitHub Bot commented on KAFKA-5602:
---

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/3539

KAFKA-5602: ducker-ak: support --custom-ducktape

Support a --custom-ducktape flag which allows developers to install
their own versions of ducktape into Docker images.  This is helpful for
ducktape development.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-5602

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3539


commit 19efac05cd3e0517f7a6ded8cfec0d38e0c0d087
Author: Colin P. Mccabe 
Date:   2017-07-17T17:04:58Z

KAFKA-5602: ducker-ak: support --custom-ducktape

Support a --custom-ducktape flag which allows developers to install
their own versions of ducktape into Docker images.  This is helpful for
ducktape development.




> Add --custom-ducktape flag to ducker-ak
> ---
>
> Key: KAFKA-5602
> URL: https://issues.apache.org/jira/browse/KAFKA-5602
> Project: Kafka
>  Issue Type: Improvement
>  Components: system tests
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>
> We should add a --custom-ducktape flag to ducker-ak.  This will make it easy 
> to run new versions of Ducktape against Apache Kafka.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5602) Add --custom-ducktape flag to ducker-ak

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-5602:
--

 Summary: Add --custom-ducktape flag to ducker-ak
 Key: KAFKA-5602
 URL: https://issues.apache.org/jira/browse/KAFKA-5602
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Colin P. McCabe
Assignee: Colin P. McCabe


We should add a --custom-ducktape flag to ducker-ak.  This will make it easy to 
run new versions of Ducktape against Apache Kafka.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4794) Add access to OffsetStorageReader from SourceConnector

[Oleg Kuznetsov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090034#comment-16090034
 ] 

Oleg Kuznetsov commented on KAFKA-4794:
---

[~fhussonnois] I meant who could give it?

> Add access to OffsetStorageReader from SourceConnector
> --
>
> Key: KAFKA-4794
> URL: https://issues.apache.org/jira/browse/KAFKA-4794
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Florian Hussonnois
>Priority: Minor
>  Labels: needs-kip
>
> Currently the offsets storage is only accessible from SourceTask to able to 
> initialize properly tasks after a restart, a crash or a reconfiguration 
> request.
> To implement more complex connectors that need to track the progression of 
> each task it would helpful to have access to an OffsetStorageReader instance 
> from the SourceConnector.
> In that way, we could have a background thread that could request a tasks 
> reconfiguration based on source offsets.
> This improvement proposal comes from a customer project that needs to 
> periodically scan directories on a shared storage for detecting and for 
> streaming new files into Kafka.
> The connector implementation is pretty straightforward.
> The connector uses a background thread to periodically scan directories. When 
> new inputs files are detected a tasks reconfiguration is requested. Then the 
> connector assigns a file subset to each task. 
> Each task stores sources offsets for the last sent record. The source offsets 
> data are:
>  - the size of file
>  - the bytes offset
>  - the bytes size 
> Tasks become idle when the assigned files are completed (in : 
> recordBytesOffsets + recordBytesSize = fileBytesSize).
> Then, the connector should be able to track offsets for each assigned file. 
> When all tasks has finished the connector can stop them or assigned new files 
> by requesting tasks reconfiguration. 
> Moreover, another advantage of monitoring source offsets from the connector 
> is detect slow or failed tasks and if necessary to be able to restart all 
> tasks.
> If you think this improvement is OK, I can work a pull request.
> Thanks,



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-4794) Add access to OffsetStorageReader from SourceConnector

[Florian Hussonnois (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090023#comment-16090023
 ] 

Florian Hussonnois commented on KAFKA-4794:
---

[~olkuznsmith] The KIP is created 
[here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector]
Currently, I've got no feedback regarding this proposal.

> Add access to OffsetStorageReader from SourceConnector
> --
>
> Key: KAFKA-4794
> URL: https://issues.apache.org/jira/browse/KAFKA-4794
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Florian Hussonnois
>Priority: Minor
>  Labels: needs-kip
>
> Currently the offsets storage is only accessible from SourceTask to able to 
> initialize properly tasks after a restart, a crash or a reconfiguration 
> request.
> To implement more complex connectors that need to track the progression of 
> each task it would helpful to have access to an OffsetStorageReader instance 
> from the SourceConnector.
> In that way, we could have a background thread that could request a tasks 
> reconfiguration based on source offsets.
> This improvement proposal comes from a customer project that needs to 
> periodically scan directories on a shared storage for detecting and for 
> streaming new files into Kafka.
> The connector implementation is pretty straightforward.
> The connector uses a background thread to periodically scan directories. When 
> new inputs files are detected a tasks reconfiguration is requested. Then the 
> connector assigns a file subset to each task. 
> Each task stores sources offsets for the last sent record. The source offsets 
> data are:
>  - the size of file
>  - the bytes offset
>  - the bytes size 
> Tasks become idle when the assigned files are completed (in : 
> recordBytesOffsets + recordBytesSize = fileBytesSize).
> Then, the connector should be able to track offsets for each assigned file. 
> When all tasks has finished the connector can stop them or assigned new files 
> by requesting tasks reconfiguration. 
> Moreover, another advantage of monitoring source offsets from the connector 
> is detect slow or failed tasks and if necessary to be able to restart all 
> tasks.
> If you think this improvement is OK, I can work a pull request.
> Thanks,



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-1694) KIP-4: Command line and centralized operations

[Stevo Slavic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090014#comment-16090014
 ] 

Stevo Slavic commented on KAFKA-1694:
-

Here's one use case: with auto topic creation disabled, when trying to publish 
with KafkaProducer to a topic that doesn't exist, one will get a timeout back - 
only way to differentiate is the timeout caused by topic does not exist 
condition or any other issue I do now by issuing topicExists check as followup 
to failed on timeout publish request.

Since auto topic creation can be disabled, all CRUD operations should be 
supported as minimum through client APIs. Yes, this kind of check can already 
be done with list support and in memory exist check. I see topicExists check as 
an optimization. With lots of topics, it's handy to have this optimization 
available in Kafka's Java client APIs, and should be easy to support so please 
consider adding it to Broker API and Java client.

> KIP-4: Command line and centralized operations
> --
>
> Key: KAFKA-1694
> URL: https://issues.apache.org/jira/browse/KAFKA-1694
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joe Stein
>Assignee: Grant Henke
>Priority: Critical
> Attachments: KAFKA-1694_2014-12-24_21:21:51.patch, 
> KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, 
> KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, 
> KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1694_2015-03-12_13:04:37.patch, 
> KAFKA-1694.patch, KAFKA-1772_1802_1775_1774_v2.patch
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5601) Refactor ReassignPartitionsCommand to use AdminClient

[Tom Bentley (JIRA)]

Tom Bentley created KAFKA-5601:
--

 Summary: Refactor ReassignPartitionsCommand to use AdminClient
 Key: KAFKA-5601
 URL: https://issues.apache.org/jira/browse/KAFKA-5601
 Project: Kafka
  Issue Type: Improvement
Reporter: Tom Bentley
Assignee: Tom Bentley


Currently the {{ReassignPartitionsCommand}} (used by 
{{kafka-reassign-partitions.sh}}) talks directly to ZooKeeper. It would be 
better to have it use the AdminClient API instead. 

This would entail creating two new protocol APIs, one to initiate the request 
and another to request the status of an in-progress reassignment. As such this 
would require a KIP.

This touches on the work of KIP-166, but that proposes to use the 
{{ReassignPartitionsCommand}} API, so should not be affected so long as that 
API is maintained. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5274) Review and improve AdminClient Javadoc for the first release (KIP-117)

[Stevo Slavic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089993#comment-16089993
 ] 

Stevo Slavic commented on KAFKA-5274:
-

Sorry for late feedback, missed that this client was being exposed and included 
in 0.11.

>From JavaDoc it's not clear with which exceptions will returned Futures 
>fail/complete in different error cases.
E.g. I wanted to migrate logic using old topic deletion in Scala AdminUtils to 
new Java AdminClient and it's not clear whether future will complete with 
success or fail with TopicDeletionInProgressException if topic deletion has 
already been requested.

Also, it seems cumbersome to force collections e.g. set of topics to be 
deleted, and map in response when all I want to do is create single topic or 
delete single topic. Please overload methods to support single item ops, 
wrap/reuse boilerplate code internally.

Was thinking of using {{all}} instead of {{values}} to avoid dealing with Map 
when deleting just single topic, but then was not sure what will be exception 
in failure even if just single delete is issued - will it be some generic 
exception saying that some of the deletion request failed, and it won't even 
have any of the exceptions of individual deletion requests, or will the all 
future on failure include just some exception from any (first) of the failed 
deletions.

In general, are there plans to get away from java.util Future and use [Flow 
once available in Java 
9|http://cs.oswego.edu/pipermail/concurrency-interest/2015-January/013641.html]?
 Future API in Java as it is now is horrible. What was rational to use Java's 
Future in client API? I don't see it in KIP-4 wiki. Maybe it's broader decision 
for all Kafka Java APIs. I'd prefer RxJava implementation and 
http://www.reactive-streams.org/ interfaces over "standard" Future API anytime 
in Java 8, even Java itself is going in that direction.

> Review and improve AdminClient Javadoc for the first release (KIP-117)
> --
>
> Key: KAFKA-5274
> URL: https://issues.apache.org/jira/browse/KAFKA-5274
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0, 0.11.1.0
>
>
> Once all the AdminClient pieces are in, we should take a pass at the Javadoc 
> and improve it wherever possible.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5588) ConsoleConsumer : uselss --new-consumer option

[Paolo Patierno (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089819#comment-16089819
 ] 

Paolo Patierno commented on KAFKA-5588:
---

I have just written the 
[KIP-176|https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+in+ConsoleConsumer+tool]
 for this and I'm going to start the discussion in the dev mailing list.

> ConsoleConsumer : uselss --new-consumer option
> --
>
> Key: KAFKA-5588
> URL: https://issues.apache.org/jira/browse/KAFKA-5588
> Project: Kafka
>  Issue Type: Bug
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Minor
>
> Hi,
> it seems to me that the --new-consumer option on the ConsoleConsumer is 
> useless.
> The useOldConsumer var is related to specify --zookeeper on the command line 
> but then the bootstrap-server option (or the --new-consumer) can't be 
> used.
> If you use --bootstrap-server option then the new consumer is used 
> automatically so no need for --new-consumer.
> It turns out the using the old or new consumer is just related on using 
> --zookeeper or --bootstrap-server option (which can't be used together, so I 
> can't use new consumer connecting to zookeeper).
> It's also clear when you use --zookeeper for the old consumer and the output 
> from help says :
> "Consider using the new consumer by passing [bootstrap-server] instead of 
> [zookeeper]"
> I'm going to remove the --new-consumer option from the tool.
> Thanks,
> Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-1694) KIP-4: Command line and centralized operations

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089810#comment-16089810
 ] 

Ismael Juma commented on KAFKA-1694:


[~sslavic], `AdminUtils` is communicating with ZooKeeper directly. The broker 
doesn't currently expose a protocol API that just checks if a topic exists. We 
are currently planning to make `listTopics` more efficient, but no immediate 
plans for a topic exists protocol API. Is there a use case where this is 
important?

> KIP-4: Command line and centralized operations
> --
>
> Key: KAFKA-1694
> URL: https://issues.apache.org/jira/browse/KAFKA-1694
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joe Stein
>Assignee: Grant Henke
>Priority: Critical
> Attachments: KAFKA-1694_2014-12-24_21:21:51.patch, 
> KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, 
> KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, 
> KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1694_2015-03-12_13:04:37.patch, 
> KAFKA-1694.patch, KAFKA-1772_1802_1775_1774_v2.patch
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-1694) KIP-4: Command line and centralized operations

[Stevo Slavic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089801#comment-16089801
 ] 

Stevo Slavic commented on KAFKA-1694:
-

Thanks for providing Java AdminClient. While migrating from Scala 
{{AdminUtils}} to Java {{AdminClient}} I noticed that in the new Java one there 
is no {{topicExists}} API support other than listing all topics and doing the 
search locally in memory.

Are there plans already and if not would it make sense to add {{topicExists}} 
to the Java AdminClient and matching Kafka broker API?

> KIP-4: Command line and centralized operations
> --
>
> Key: KAFKA-1694
> URL: https://issues.apache.org/jira/browse/KAFKA-1694
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joe Stein
>Assignee: Grant Henke
>Priority: Critical
> Attachments: KAFKA-1694_2014-12-24_21:21:51.patch, 
> KAFKA-1694_2015-01-12_15:28:41.patch, KAFKA-1694_2015-01-12_18:54:48.patch, 
> KAFKA-1694_2015-01-13_19:30:11.patch, KAFKA-1694_2015-01-14_15:42:12.patch, 
> KAFKA-1694_2015-01-14_18:07:39.patch, KAFKA-1694_2015-03-12_13:04:37.patch, 
> KAFKA-1694.patch, KAFKA-1772_1802_1775_1774_v2.patch
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5600) GroupMetadataManager doesn't read offsets of segmented logs correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089744#comment-16089744
 ] 

ASF GitHub Bot commented on KAFKA-5600:
---

GitHub user bjrke opened a pull request:

https://github.com/apache/kafka/pull/3538

KAFKA-5600: fix GroupMetadataManager doesn't read offsets of segmente…

…d log correctly

the while loop was too big and need to be closed earlier
to see the fix, ignore whitespace since most of it is indentation

this bug was introduced by commit
5bd06f1d542e6b588a1d402d059bc24690017d32

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justsocialapps/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3538


commit 6b625780dd67c1dc04a90c58f99f77efd18b9d29
Author: Jan Burkhardt 
Date:   2017-07-14T14:30:36Z

KAFKA-5600: fix GroupMetadataManager doesn't read offsets of segmented log 
correctly

the while loop was too big and need to be closed earlier
to see the fix, ignore whitespace since most of it is indentation

this bug was introduced by commit
5bd06f1d542e6b588a1d402d059bc24690017d32




> GroupMetadataManager doesn't read offsets of segmented logs correctly
> -
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
>Priority: Critical
>  Labels: reliability
> Fix For: 0.11.0.1
>
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets, consumers will start at a wrong position when 
> metadata and offset events are in both segments.
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
> 5mins). The close of the consumer is needed to have metadata events in the 
> segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
> commit 
> https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32
> I will prepare a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read offsets of segmented logs correctly

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5600:
---
Labels: reliability  (was: )

> GroupMetadataManager doesn't read offsets of segmented logs correctly
> -
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
>  Labels: reliability
> Fix For: 0.11.0.1
>
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets, consumers will start at a wrong position when 
> metadata and offset events are in both segments.
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
> 5mins). The close of the consumer is needed to have metadata events in the 
> segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
> commit 
> https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32
> I will prepare a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read offsets of segmented logs correctly

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5600:
---
Fix Version/s: 0.11.0.1

> GroupMetadataManager doesn't read offsets of segmented logs correctly
> -
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
>  Labels: reliability
> Fix For: 0.11.0.1
>
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets, consumers will start at a wrong position when 
> metadata and offset events are in both segments.
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
> 5mins). The close of the consumer is needed to have metadata events in the 
> segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
> commit 
> https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32
> I will prepare a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read offsets of segmented logs correctly

[Jan Burkhardt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Burkhardt updated KAFKA-5600:
-
Description: 
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets, consumers will start at a wrong position when metadata 
and offset events are in both segments.

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
/tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
5mins). The close of the consumer is needed to have metadata events in the 
segments too.
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32
I will prepare a pull request.


  was:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets, consumers will start at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
/tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
5mins). The close of the consumer is needed to have metadata events in the 
segments too.
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32
I will prepare a pull request.



> GroupMetadataManager doesn't read offsets of segmented logs correctly
> -
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets, consumers will start at a wrong position when 
> metadata and offset events are in both segments.
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
> 5mins). The close of the consumer is needed to have metadata events in the 
> segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the 

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read Offsets of segmented log correctly

[Jan Burkhardt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Burkhardt updated KAFKA-5600:
-
Description: 
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets, consumers will start at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
/tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
5mins). The close of the consumer is needed to have metadata events in the 
segments too.
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32



  was:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets, consumers will start at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
/tmp/kafka-logs/__consumer_offsets-27. This step takes some time. The close of 
the consumer is needed to have metadata events in the segments too.
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32




> GroupMetadataManager doesn't read Offsets of segmented log correctly
> 
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets, consumers will start at a wrong position
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 
> 5mins). The close of the consumer is needed to have metadata events in the 
> segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong 

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read Offsets of segmented log correctly

[Jan Burkhardt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Burkhardt updated KAFKA-5600:
-
Description: 
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets, consumers will start at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
/tmp/kafka-logs/__consumer_offsets-27. This step takes some time. The close of 
the consumer is needed to have metadata events in the segments too.
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32



  was:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets, consumers will start at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
__consumer_offsets-27. This step takes some time. The close of the consumer is 
needed to also have metadata events in the segments
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32




> GroupMetadataManager doesn't read Offsets of segmented log correctly
> 
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets, consumers will start at a wrong position
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time. The 
> close of the consumer is needed to have metadata events in the segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
> commit 
> 

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read Offsets of segmented log correctly

[Jan Burkhardt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Burkhardt updated KAFKA-5600:
-
Description: 
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets consumer will start at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
__consumer_offsets-27. This step takes some time. The close of the consumer is 
needed to also have metadata events in the segments
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32



  was:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets it starts at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
__consumer_offsets-27. This step takes some time. The close of the consumer is 
needed to also have metadata events in the segments
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 
https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32




> GroupMetadataManager doesn't read Offsets of segmented log correctly
> 
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets consumer will start at a wrong position
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in __consumer_offsets-27. This step takes some time. The close of the 
> consumer is needed to also have metadata events in the segments
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
> commit 
> 

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read Offsets of segmented log correctly

[Jan Burkhardt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Burkhardt updated KAFKA-5600:
-
Description: 
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets it starts at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
__consumer_offsets-27. This step takes some time. The close of the consumer is 
needed to also have metadata events in the segments
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
commit 5bd06f1d542e6b588a1d402d059bc24690017d32



  was:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets it starts at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
__consumer_offsets-27. This step takes some time. The close of the consumer is 
needed to also have metadata events in the segments
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position




> GroupMetadataManager doesn't read Offsets of segmented log correctly
> 
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets it starts at a wrong position
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in __consumer_offsets-27. This step takes some time. The close of the 
> consumer is needed to also have metadata events in the segments
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with 
> commit 5bd06f1d542e6b588a1d402d059bc24690017d32



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5585) Failover in a replicated Cluster does not work

[Thomas Bayer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Bayer resolved KAFKA-5585.
-
Resolution: Cannot Reproduce

After setting up everything from scratch I was not able to reproduce the 
failure on Mac OS.

> Failover in a replicated Cluster does not work
> --
>
> Key: KAFKA-5585
> URL: https://issues.apache.org/jira/browse/KAFKA-5585
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.0
> Environment: Linux, Mac OSX
>Reporter: Thomas Bayer
> Attachments: broker_zookeeper_configs.zip, SimpleConsumer.java, 
> SimpleProducer.java, Stress Test Windows.xlsx, test_project_files.zip
>
>
> Failover does not work in a cluster with 3 nodes and a replicated topic with 
> factor 3.
> You can reproduce it als follows: Setup 3 Kafka Nodes and 1 Zookeeper. Than 
> create a topic with factor 3. Start a consumer. Stop a node. Write to the 
> topic. Now you get warnings that the client can not connect to a broker. The 
> consumer does not receive any messages.
> The same setup works like a charm with 0.10.2.1.
> Broker Config:
> {{broker.id=1
> listeners=PLAINTEXT://:9091
> log.dirs=cluster/logs/node-1
> broker.id=2
> listeners=PLAINTEXT://:9092
> log.dirs=cluster/logs/node-2
> broker.id=3
> listeners=PLAINTEXT://:9093
> log.dirs=cluster/logs/node-3}}
> Rest of the config is from the distribution.
> Producer and consumer config: see attached files
> *Log Consumer:*
> 2017-07-12 16:15:26 WARN  ConsumerCoordinator:649 - Auto-commit of offsets 
> {produktion-0=OffsetAndMetadata{offset=10, metadata=''}} failed for group a: 
> Offset commit failed with a retriable exception. You should retry committing 
> offsets. The underlying error was: The coordinator is not available.
> 2017-07-12 16:15:26 WARN  NetworkClient:588 - Connection to node 2147483645 
> could not be established. Broker may not be available.
> 2017-07-12 16:15:26 WARN  NetworkClient:588 - Connection to node 2 could not 
> be established. Broker may not be available.
> *Log Producer:*
> {{2017-07-12 16:15:32 WARN  NetworkClient:588 - Connection to node -1 could 
> not be established. Broker may not be available.}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5585) Failover in a replicated Cluster does not work

[Thomas Bayer (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089709#comment-16089709
 ] 

Thomas Bayer commented on KAFKA-5585:
-

After repeating the whole setup from scratch it worked with version 0.11.0.0. 
Sorry for the trouble and thanks for the fast reaction!

> Failover in a replicated Cluster does not work
> --
>
> Key: KAFKA-5585
> URL: https://issues.apache.org/jira/browse/KAFKA-5585
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.0
> Environment: Linux, Mac OSX
>Reporter: Thomas Bayer
> Attachments: broker_zookeeper_configs.zip, SimpleConsumer.java, 
> SimpleProducer.java, Stress Test Windows.xlsx, test_project_files.zip
>
>
> Failover does not work in a cluster with 3 nodes and a replicated topic with 
> factor 3.
> You can reproduce it als follows: Setup 3 Kafka Nodes and 1 Zookeeper. Than 
> create a topic with factor 3. Start a consumer. Stop a node. Write to the 
> topic. Now you get warnings that the client can not connect to a broker. The 
> consumer does not receive any messages.
> The same setup works like a charm with 0.10.2.1.
> Broker Config:
> {{broker.id=1
> listeners=PLAINTEXT://:9091
> log.dirs=cluster/logs/node-1
> broker.id=2
> listeners=PLAINTEXT://:9092
> log.dirs=cluster/logs/node-2
> broker.id=3
> listeners=PLAINTEXT://:9093
> log.dirs=cluster/logs/node-3}}
> Rest of the config is from the distribution.
> Producer and consumer config: see attached files
> *Log Consumer:*
> 2017-07-12 16:15:26 WARN  ConsumerCoordinator:649 - Auto-commit of offsets 
> {produktion-0=OffsetAndMetadata{offset=10, metadata=''}} failed for group a: 
> Offset commit failed with a retriable exception. You should retry committing 
> offsets. The underlying error was: The coordinator is not available.
> 2017-07-12 16:15:26 WARN  NetworkClient:588 - Connection to node 2147483645 
> could not be established. Broker may not be available.
> 2017-07-12 16:15:26 WARN  NetworkClient:588 - Connection to node 2 could not 
> be established. Broker may not be available.
> *Log Producer:*
> {{2017-07-12 16:15:32 WARN  NetworkClient:588 - Connection to node -1 could 
> not be established. Broker may not be available.}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read Offsets of segmented log correctly

[Jan Burkhardt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Burkhardt updated KAFKA-5600:
-
Description: 
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets it starts at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
__consumer_offsets-27. This step takes some time. The close of the consumer is 
needed to also have metadata events in the segments
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position



  was:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and 
needs to reload offsets it starts at a wrong position

Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries 
one by one and then closes the consumer. This leads to a 2nd segment in 
__consumer_offsets-27
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
localhost:9092 --topic test --consumer-property group.id=testgroup
{code}

Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end

The Reason for this behavior is the closing brace of the while loop in 
GroupMetadataManager#loadGroupsAndOffsets at a wrong position




> GroupMetadataManager doesn't read Offsets of segmented log correctly
> 
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
>Reporter: Jan Burkhardt
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and 
> needs to reload offsets it starts at a wrong position
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh 
> config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh 
> config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic 
> test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 
> entries one by one and then closes the consumer. This leads to a 2nd segment 
> in __consumer_offsets-27. This step takes some time. The close of the 
> consumer is needed to also have metadata events in the segments
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server 
> localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in 
> GroupMetadataManager#loadGroupsAndOffsets at a wrong position



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5599) ConsoleConsumer : --new-consumer option as deprecated

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089684#comment-16089684
 ] 

Ismael Juma commented on KAFKA-5599:


For background, there was a time when "--new-consumer" was required to use the 
new consumer. We changed that in a recent release because we wanted the new 
consumer to become the default. The idea was to deprecated and eventually 
remove "--new-consumer" in subsequent releases.

> ConsoleConsumer : --new-consumer option as deprecated
> -
>
> Key: KAFKA-5599
> URL: https://issues.apache.org/jira/browse/KAFKA-5599
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>
> Hi,
> it seems to me that the --new-consumer option on the ConsoleConsumer is 
> useless.
> The useOldConsumer var is related to specify --zookeeper on the command line 
> but then the bootstrap-server option (or the --new-consumer) can't be 
> used.
> If you use --bootstrap-server option then the new consumer is used 
> automatically so no need for --new-consumer.
> It turns out the using the old or new consumer is just related on using 
> --zookeeper or --bootstrap-server option (which can't be used together, so I 
> can't use new consumer connecting to zookeeper).
> It's also clear when you use --zookeeper for the old consumer and the output 
> from help says :
> "Consider using the new consumer by passing [bootstrap-server] instead of 
> [zookeeper]"
> Before removing the --new-consumer option, this JIRA is for marking it as 
> deprecated in the next release (then moving for a new release on removing 
> such option).
> Thanks,
> Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-5599) ConsoleConsumer : --new-consumer option as deprecated

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089684#comment-16089684
 ] 

Ismael Juma edited comment on KAFKA-5599 at 7/17/17 11:35 AM:
--

For background, there was a time when the "new-consumer" option was required to 
use the new consumer. We changed that in a recent release because we wanted the 
new consumer to become the default. The idea was to deprecated and eventually 
remove "--new-consumer" in subsequent releases.


was (Author: ijuma):
For background, there was a time when "--new-consumer" was required to use the 
new consumer. We changed that in a recent release because we wanted the new 
consumer to become the default. The idea was to deprecated and eventually 
remove "--new-consumer" in subsequent releases.

> ConsoleConsumer : --new-consumer option as deprecated
> -
>
> Key: KAFKA-5599
> URL: https://issues.apache.org/jira/browse/KAFKA-5599
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>
> Hi,
> it seems to me that the --new-consumer option on the ConsoleConsumer is 
> useless.
> The useOldConsumer var is related to specify --zookeeper on the command line 
> but then the bootstrap-server option (or the --new-consumer) can't be 
> used.
> If you use --bootstrap-server option then the new consumer is used 
> automatically so no need for --new-consumer.
> It turns out the using the old or new consumer is just related on using 
> --zookeeper or --bootstrap-server option (which can't be used together, so I 
> can't use new consumer connecting to zookeeper).
> It's also clear when you use --zookeeper for the old consumer and the output 
> from help says :
> "Consider using the new consumer by passing [bootstrap-server] instead of 
> [zookeeper]"
> Before removing the --new-consumer option, this JIRA is for marking it as 
> deprecated in the next release (then moving for a new release on removing 
> such option).
> Thanks,
> Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5588) ConsumerConsole : uselss --new-consumer option

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089621#comment-16089621
 ] 

Ismael Juma commented on KAFKA-5588:


That's right [~vahid], we generally need a KIP for changes with a backwards 
compatibility impact.

> ConsumerConsole : uselss --new-consumer option
> --
>
> Key: KAFKA-5588
> URL: https://issues.apache.org/jira/browse/KAFKA-5588
> Project: Kafka
>  Issue Type: Bug
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Minor
>
> Hi,
> it seems to me that the --new-consumer option on the ConsoleConsumer is 
> useless.
> The useOldConsumer var is related to specify --zookeeper on the command line 
> but then the bootstrap-server option (or the --new-consumer) can't be 
> used.
> If you use --bootstrap-server option then the new consumer is used 
> automatically so no need for --new-consumer.
> It turns out the using the old or new consumer is just related on using 
> --zookeeper or --bootstrap-server option (which can't be used together, so I 
> can't use new consumer connecting to zookeeper).
> It's also clear when you use --zookeeper for the old consumer and the output 
> from help says :
> "Consider using the new consumer by passing [bootstrap-server] instead of 
> [zookeeper]"
> I'm going to remove the --new-consumer option from the tool.
> Thanks,
> Paolo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KAFKA-5547) Return topic authorization failed if no topic describe access

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5547:
---
Labels: security usability  (was: )

> Return topic authorization failed if no topic describe access
> -
>
> Key: KAFKA-5547
> URL: https://issues.apache.org/jira/browse/KAFKA-5547
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>  Labels: security, usability
> Fix For: 0.11.1.0
>
>
> We previously made a change to several of the request APIs to return 
> UNKNOWN_TOPIC_OR_PARTITION if the principal does not have Describe access to 
> the topic. The thought was to avoid leaking information about which topics 
> exist. The problem with this is that a client which sees this error will just 
> keep retrying because it is usually treated as retriable. It seems, however, 
> that we could return TOPIC_AUTHORIZATION_FAILED instead and still avoid 
> leaking information as long as we ensure that the Describe authorization 
> check comes before the topic existence check. This would avoid the ambiguity 
> on the client.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5547) Return topic authorization failed if no topic describe access

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089617#comment-16089617
 ] 

Ismael Juma commented on KAFKA-5547:


[~hachikuji], it seems like that could work. As long as we're careful to always 
return the same error code for topics that exist or don't exist when the user 
is not authorized, it should be fine. Also, we also have to make sure to 
restrict any topic regex to the authorized topics (which we do already).

> Return topic authorization failed if no topic describe access
> -
>
> Key: KAFKA-5547
> URL: https://issues.apache.org/jira/browse/KAFKA-5547
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>
> We previously made a change to several of the request APIs to return 
> UNKNOWN_TOPIC_OR_PARTITION if the principal does not have Describe access to 
> the topic. The thought was to avoid leaking information about which topics 
> exist. The problem with this is that a client which sees this error will just 
> keep retrying because it is usually treated as retriable. It seems, however, 
> that we could return TOPIC_AUTHORIZATION_FAILED instead and still avoid 
> leaking information as long as we ensure that the Describe authorization 
> check comes before the topic existence check. This would avoid the ambiguity 
> on the client.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-3856) Cleanup Kafka Streams builder API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089507#comment-16089507
 ] 

ASF GitHub Bot commented on KAFKA-3856:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/3536

KAFKA-3856: Refactoring for KIP-120

 - extract InternalTopologyBuilder from TopologyBuilder
 - deprecate all "leaking" methods from public TopologyBuilder API

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-3856-extract-internal-topology-builder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3536


commit b963505f525d0ebce40ae90b921701ab2224f796
Author: Matthias J. Sax 
Date:   2017-07-13T05:14:46Z

KAFKA-3856: Refactoring for KIP-120
 - extract InternalTopologyBuilder from TopologyBuilder
 - deprecate all "leaking" methods from public TopologyBuilder API




> Cleanup Kafka Streams builder API
> -
>
> Key: KAFKA-3856
> URL: https://issues.apache.org/jira/browse/KAFKA-3856
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Matthias J. Sax
>  Labels: api, kip
> Fix For: 0.11.1.0
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-120%3A+Cleanup+Kafka+Streams+builder+API



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)