[jira] [Commented] (KAFKA-13351) Add possibility to write kafka headers in Kafka Console Producer

[Seweryn Habdank-Wojewodzki (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438573#comment-17438573
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-13351:


Thanks a lot!

> Add possibility to write kafka headers in Kafka Console Producer
> 
>
> Key: KAFKA-13351
> URL: https://issues.apache.org/jira/browse/KAFKA-13351
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 2.8.1
>Reporter: Seweryn Habdank-Wojewodzki
>Assignee: Florin Akermann
>Priority: Major
>
> Dears,
> Currently there is an asymetry between Kafka Console Consumer and Producer.
> Kafka Consumer can display headers (KAFKA-6733), but Kafka Producer cannot 
> produce them.
> It would be good to unify this and add possibility to Kafka Console Producer 
> to produce them.
> Similar ticket is here: KAFKA-6574, but it is very old and does not 
> represents current state of the software.
> Please consider this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13389) Add to kafka shell scripts checks about server state

[Seweryn Habdank-Wojewodzki (Jira)]

Seweryn Habdank-Wojewodzki created KAFKA-13389:
--

 Summary: Add to kafka shell scripts checks about server state
 Key: KAFKA-13389
 URL: https://issues.apache.org/jira/browse/KAFKA-13389
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: Seweryn Habdank-Wojewodzki


Hello,

Within the discussion with Confluent included in the Confluent Support Ticket: 
[#71907|https://urldefense.com/v3/__https:/support.confluent.io/hc/requests/71907__;!!OMGRPR5eiCE28w!9-OfZd3vUrXgjEtagEYeB1O5tmebDaANKfi6c-VRV0RrdcFEnFlzb7pDwpSwJTZ7qFnigilEAPhGW1vS5XdsSkU$],
 we found out that according to "Eventually Consistency" Kafka shell scripts 
may deliver wrong information, for example when listing topics, the result 
might be empty even if topics are existing, but Server status is not in synch 
(e.g. when URP > 0).

To be concrete. This call below may return empty list, if server is not in 
synch.

{code}

$ ./bin/kafka-topics.sh --bootstrap-server= 
--list

{code}

 

Remark from Confluent engineers is: that before getting whose results, one have 
to check server status and in particular URP shall be 0, otherwise results 
might be wrong.

So in fact Kafka shell scripts contains bug delivering possibly broken results 
and not reporting error instead.

The proposal here is to add to all Kafka shell scripts check if server status 
is proper (e.g. URP is 0) and in case of having server not in good state, 
instead of returning possible wrong values, script shall return proper error 
code with message, that server is not in proper state.

Why in Kafka shell scripts and not on the user side?

Because Kafka Team knows all server conditions and can describe server status 
much better than any other user and checks will be done centrally for all 
users, who do not need to always implement the same. Also updates, when Kafka 
changes own API will be done synchronously.

 

Thanks in advance for adding those checks and best regards,

Seweryn.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki resolved KAFKA-7214.
---
Resolution: Workaround

The solution is to avoid low values of {{max.block.ms}}

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1, 2.3.0, 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: qns-1.1.zip, qns-1.zip
>
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-7214.
-

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1, 2.3.0, 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: qns-1.1.zip, qns-1.zip
>
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-6777.
-

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (KAFKA-6882) Wrong producer settings may lead to DoS on Kafka Server

[Seweryn Habdank-Wojewodzki (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-6882.
-

> Wrong producer settings may lead to DoS on Kafka Server
> ---
>
> Key: KAFKA-6882
> URL: https://issues.apache.org/jira/browse/KAFKA-6882
> Project: Kafka
>  Issue Type: Bug
>  Components: core, producer 
>Affects Versions: 1.0.1, 1.1.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> The documentation of the following parameters “linger.ms” and “batch.size” is 
> a bit confusing. In fact those parameters wrongly set on the producer side 
> might completely destroy BROKER throughput.
> I see, that smart developers are reading documentation of those parameters.
> Then they want to have super performance and super safety, so they set 
> something like this below:
> {code}
> kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
> kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
> {code}
> Then we have situation, when each and every message is send separately. 
> TCP/IP protocol is really busy in that case and when they needed high 
> throughput they got much less throughput, as every message is goes separately 
> causing all network communication and TCP/IP overhead significant.
> Those settings are good only if someone sends critical messages like once a 
> while (e.g. one message per minute) and not when throughput is important by 
> sending thousands messages per second.
> Situation is even worse when smart developers are reading, that for safety, 
> they need acknowledges from all cluster nodes. So they are adding:
> {code}
> kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
> {code}
> And this is the end of Kafka performance! 
> Even worse it is not a problem for the Kafka producer. The problem remains at 
> the server (cluster, broker) side. The server is so busy by acknowledging 
> *each and every* message from all nodes, that other work is NOT performed, so 
> the end to end performance is almost none.
> I would like to ask you to improve documentation of this parameters.
> And consider corner cases is case of providing detailed information how 
> extreme values of parameters - namely lowest and highest – may influence work 
> of the cluster.
> This was documentation issue. 
> On the other hand it is security/safety matter.
> Technically the problem is that __commit_offsets topic is loaded with 
> enormous amount of messages. It leads to the situation, when Kafka Broker is 
> exposed to *DoS *due to the Producer settings. Three lines of code a bit load 
> and the Kafka cluster is dead.
> I suppose there are ways to prevent such a situation on the cluster side, but 
> it require some logic to be implemented to detect such a simple but efficient 
> DoS.
> BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
> the other producer makes problems?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (KAFKA-8548) Inconsistency in Kafka Documentation

[Seweryn Habdank-Wojewodzki (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-8548.
-

> Inconsistency in Kafka Documentation
> 
>
> Key: KAFKA-8548
> URL: https://issues.apache.org/jira/browse/KAFKA-8548
> Project: Kafka
>  Issue Type: Task
>  Components: documentation
>Affects Versions: 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Minor
>
> Dears,
> Two parts (referenced below) of [documentation 
> |http://kafka.apache.org/documentation/] are not quite consistent.
> In one text we can read, that max.poll.interval.ms has defaut value 
> Integer.MAX_VALUE, in the other it is 300 000.
> Part 1.
> {quote}
> The default values for two configurations of the StreamsConfig class were 
> changed to improve the resiliency of Kafka Streams applications. The internal 
> Kafka Streams producer retries default value was changed from 0 to 10. The 
> internal Kafka Streams consumer max.poll.interval.ms default value was 
> changed from 30 to {color:#FF}Integer.MAX_VALUE{color}.
> {quote}
>  
> Part 2. - Table
> |max.poll.interval.ms|The maximum delay between invocations of poll() when 
> using consumer group management. This places an upper bound on the amount of 
> time that the consumer can be idle before fetching more records. If poll() is 
> not called before expiration of this timeout, then the consumer is considered 
> failed and the group will rebalance in order to reassign the partitions to 
> another member.|int|{color:#FF}30{color}|[1,...]|medium|
> Which value is then default :-)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (KAFKA-9221) Kafka REST Proxy wrongly converts quotes in message when sending json

[Seweryn Habdank-Wojewodzki (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-9221.
-

> Kafka REST Proxy wrongly converts quotes in message when sending json
> -
>
> Key: KAFKA-9221
> URL: https://issues.apache.org/jira/browse/KAFKA-9221
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 2.3.0
> Environment: Linux redhat
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Kafka REST Proxy has a problem when sending/converting json files (e.g. 
> json.new) into Kafka protocol. For example JSON file:
> {code:java}
> {"records":[{"value":"rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
>  1337 1572276922"}]}
> {code}
> is sent using call to Kafka REST Proxy on localhost:8073:
> {code:java}
> curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -H 
> "Accept: application/vnd.kafka.v2+json" --data @json.new  
> http://localhost:8073/topics/somple_topic -i 
> {code}
> in Kafka in some_topic we got:
> {code:java}
> "rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
>  1337 1572276922"
> {code}
> but expected is that message has no quotes:
> {code:java}
> rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
>  1337 1572276922
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13351) Add possibility to write kafka headers in Kafka Console Producer

[Seweryn Habdank-Wojewodzki (Jira)]

Seweryn Habdank-Wojewodzki created KAFKA-13351:
--

 Summary: Add possibility to write kafka headers in Kafka Console 
Producer
 Key: KAFKA-13351
 URL: https://issues.apache.org/jira/browse/KAFKA-13351
 Project: Kafka
  Issue Type: Wish
Affects Versions: 2.8.1
Reporter: Seweryn Habdank-Wojewodzki


Dears,

Currently there is an asymetry between Kafka Console Consumer and Producer.
Kafka Consumer can display headers (KAFKA-6733), but Kafka Producer cannot 
produce them.

It would be good to unify this and add possibility to Kafka Console Producer to 
produce them.

Similar ticket is here: KAFKA-6574, but it is very old and does not represents 
current state of the software.

Please consider this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9221) Kafka REST Proxy wrongly converts quotes in message when sending json

[Seweryn Habdank-Wojewodzki (Jira)]

Seweryn Habdank-Wojewodzki created KAFKA-9221:
-

 Summary: Kafka REST Proxy wrongly converts quotes in message when 
sending json
 Key: KAFKA-9221
 URL: https://issues.apache.org/jira/browse/KAFKA-9221
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 2.3.0
 Environment: Linux redhat
Reporter: Seweryn Habdank-Wojewodzki


Kafka REST Proxy has a problem when sending/converting json files (e.g. 
json.new) into Kafka protocol. For example JSON file:
{code:java}
{"records":[{"value":"rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
 1337 1572276922"}]}
{code}
is sent using call to Kafka REST Proxy on localhost:8073:
{code:java}
curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" -H "Accept: 
application/vnd.kafka.v2+json" --data @json.new  
http://localhost:8073/topics/somple_topic -i 
{code}
in Kafka in some_topic we got:
{code:java}
"rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
 1337 1572276922"
{code}
but expected is that message has no quotes:
{code:java}
rest.kafka.testmetric,host=server.com,partition=8,topic=my_topic,url=http:--localhost:7071-metrics
 1337 1572276922
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883690#comment-16883690
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7214:
---

I have found the problem. It is following.

If the value of the _max.block.ms_ is too small in comparison to what is doable 
in client server connection, then streamming API after getting problems to 
obtain meta data from broker is not trying to do it more times, but only throws 
exception and ended the life of the stream.

Finally I observe this in two situations:
# When client really lost connection to the broker, but was not intended to 
lose it. For example network downtime.
# When broker is so much loaded with other work, that will not respond to the 
streamming client. Mostly this was my case.

IMHO this is bug. Streamming API shall not give up - or the number of retries 
to obtain metadata shall be configurable. This is different to the number of 
retries to send the message. Metadata message from logical point of view has 
completely different function than normal messages.

To reproduce it, one can set this _max.block.ms_ to low value like 10-100ms and 
start streamming app connected to broker, which is loaded or disconnect 
connection after start.

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1, 2.3.0, 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: qns-1.1.zip, qns-1.zip
>
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-7214:
--
Affects Version/s: 2.3.0

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1, 2.3.0, 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: qns-1.1.zip, qns-1.zip
>
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Resolved] (KAFKA-6882) Wrong producer settings may lead to DoS on Kafka Server

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki resolved KAFKA-6882.
---
Resolution: Won't Fix

As there are no improvment proposals I am closing it. :-)

> Wrong producer settings may lead to DoS on Kafka Server
> ---
>
> Key: KAFKA-6882
> URL: https://issues.apache.org/jira/browse/KAFKA-6882
> Project: Kafka
>  Issue Type: Bug
>  Components: core, producer 
>Affects Versions: 1.0.1, 1.1.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> The documentation of the following parameters “linger.ms” and “batch.size” is 
> a bit confusing. In fact those parameters wrongly set on the producer side 
> might completely destroy BROKER throughput.
> I see, that smart developers are reading documentation of those parameters.
> Then they want to have super performance and super safety, so they set 
> something like this below:
> {code}
> kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
> kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
> {code}
> Then we have situation, when each and every message is send separately. 
> TCP/IP protocol is really busy in that case and when they needed high 
> throughput they got much less throughput, as every message is goes separately 
> causing all network communication and TCP/IP overhead significant.
> Those settings are good only if someone sends critical messages like once a 
> while (e.g. one message per minute) and not when throughput is important by 
> sending thousands messages per second.
> Situation is even worse when smart developers are reading, that for safety, 
> they need acknowledges from all cluster nodes. So they are adding:
> {code}
> kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
> {code}
> And this is the end of Kafka performance! 
> Even worse it is not a problem for the Kafka producer. The problem remains at 
> the server (cluster, broker) side. The server is so busy by acknowledging 
> *each and every* message from all nodes, that other work is NOT performed, so 
> the end to end performance is almost none.
> I would like to ask you to improve documentation of this parameters.
> And consider corner cases is case of providing detailed information how 
> extreme values of parameters - namely lowest and highest – may influence work 
> of the cluster.
> This was documentation issue. 
> On the other hand it is security/safety matter.
> Technically the problem is that __commit_offsets topic is loaded with 
> enormous amount of messages. It leads to the situation, when Kafka Broker is 
> exposed to *DoS *due to the Producer settings. Three lines of code a bit load 
> and the Kafka cluster is dead.
> I suppose there are ways to prevent such a situation on the cluster side, but 
> it require some logic to be implemented to detect such a simple but efficient 
> DoS.
> BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
> the other producer makes problems?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (KAFKA-7375) Improve error messages verbosity

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-7375:
--
Affects Version/s: 2.2.1

> Improve error messages verbosity
> 
>
> Key: KAFKA-7375
> URL: https://issues.apache.org/jira/browse/KAFKA-7375
> Project: Kafka
>  Issue Type: Task
>Affects Versions: 1.1.1, 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> Very often when clients are trying to connect we see in Kafka logs:
> {code}
> “org.apache.kafka.common.network.SslTransportLayer  - Failed to send SSL 
> Close message“
> {code}
> The problem here is following: there is no word who? No IP, no client addres, 
> nothing.
> Would be great to have in all error or warning reports like this one, very 
> precize information which client has a problem, to be able to solve it. When 
> the number of clients is more than 10, this message is completely useless and 
> when there are even more clients it really spams logs.
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-7214:
--
Affects Version/s: 2.2.1

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1, 2.2.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: qns-1.1.zip, qns-1.zip
>
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8548) Inconsistency in Kafka Documentation

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-8548:
-

 Summary: Inconsistency in Kafka Documentation
 Key: KAFKA-8548
 URL: https://issues.apache.org/jira/browse/KAFKA-8548
 Project: Kafka
  Issue Type: Task
  Components: documentation
Affects Versions: 2.2.1
Reporter: Seweryn Habdank-Wojewodzki


Dears,

Two parts (referenced below) of [documentation 
|http://kafka.apache.org/documentation/] are not quite consistent.

In one text we can read, that max.poll.interval.ms has defaut value 
Integer.MAX_VALUE, in the other it is 300 000.

Part 1.

{quote}
The default values for two configurations of the StreamsConfig class were 
changed to improve the resiliency of Kafka Streams applications. The internal 
Kafka Streams producer retries default value was changed from 0 to 10. The 
internal Kafka Streams consumer max.poll.interval.ms default value was changed 
from 30 to {color:#FF}Integer.MAX_VALUE{color}.
{quote}
 
Part 2. - Table

|max.poll.interval.ms|The maximum delay between invocations of poll() when 
using consumer group management. This places an upper bound on the amount of 
time that the consumer can be idle before fetching more records. If poll() is 
not called before expiration of this timeout, then the consumer is considered 
failed and the group will rebalance in order to reassign the partitions to 
another member.|int|{color:#FF}30{color}|[1,...]|medium|

Which value is then default :-)




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KAFKA-4849) Bug in KafkaStreams documentation

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-4849.
-

> Bug in KafkaStreams documentation
> -
>
> Key: KAFKA-4849
> URL: https://issues.apache.org/jira/browse/KAFKA-4849
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Seweryn Habdank-Wojewodzki
>Assignee: Matthias J. Sax
>Priority: Minor
>
> At the page: https://kafka.apache.org/documentation/streams
>  
> In the chapter titled Application Configuration and Execution, in the example 
> there is a line:
>  
> settings.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "zookeeper1:2181");
>  
> but ZOOKEEPER_CONNECT_CONFIG is deprecated in the Kafka version 0.10.2.0.
>  
> Also the table on the page: 
> https://kafka.apache.org/0102/documentation/#streamsconfigs is a bit 
> misleading.
> 1. Again zookeeper.connect is deprecated.
> 2. The client.id and zookeeper.connect are marked by high importance, 
> but according to http://docs.confluent.io/3.2.0/streams/developer-guide.html 
> none of them are important to initialize the stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-6699.
-

> When one of two Kafka nodes are dead, streaming API cannot handle messaging
> ---
>
> Key: KAFKA-6699
> URL: https://issues.apache.org/jira/browse/KAFKA-6699
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> I am observing quite often, when Kafka Broker is partly dead(*), then 
> application, which uses streaming API are doing nothing.
> (*) Partly dead in my case it means that one of two Kafka nodes are out of 
> order. 
> Especially when disk is full on one machine, then Broker is going in some 
> strange state, where streaming API goes vacations. It seems like regular 
> producer/consumer API has no problem in such a case.
> Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki resolved KAFKA-6777.
---
Resolution: Won't Fix

Last comment is accepted. We have to prepare other measures to mitigate this 
situation.
I am resolving the ticket :-).


> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612374#comment-16612374
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/12/18 3:45 PM:


[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when 
they need to handle the case below?

{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled 
exception: Exception caught in process. taskId=0_2, 
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2, 
offset=1892533025; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
 in thread 
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State 
transition from RUNNING to PENDING_SHUTDOWN
{code}

* What is wrong there? 
* What are the steps to avoid this in the future? 
* How to repair the situation?



was (Author: habdank):
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when 
they need to handle the case below?

{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled 
exception: Exception caught in process. taskId=0_2, 
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2, 
offset=1892533025; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
 in thread 
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State 
transition from RUNNING to PENDING_SHUTDOWN
{code}

What is wrong there? What are the steps to avoid this in the future? How to 
repair the situation?


> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn 

[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612374#comment-16612374
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/12/18 3:45 PM:


[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when 
they need to handle the case below?

{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled 
exception: Exception caught in process. taskId=0_2, 
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2, 
offset=1892533025; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
 in thread 
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State 
transition from RUNNING to PENDING_SHUTDOWN
{code}

What is wrong there? What are the steps to avoid this in the future? How to 
repair the situation?



was (Author: habdank):
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when 
they need to handle the case below?

{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled 
exception: Exception caught in process. taskId=0_2, 
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2, 
offset=1892533025; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
 in thread 
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State 
transition from RUNNING to PENDING_SHUTDOWN
{code}

What is wrong there? What are the steps to avoid this in the future?


> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> 

[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612374#comment-16612374
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/12/18 3:44 PM:


[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when 
they need to handle the case below?

{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled 
exception: Exception caught in process. taskId=0_2, 
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2, 
offset=1892533025; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
 in thread 
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State 
transition from RUNNING to PENDING_SHUTDOWN
{code}

What is wrong there? What are the steps to avoid this in the future?



was (Author: habdank):
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when 
they need to handle the case below?

{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled 
exception: Exception caught in process. taskId=0_2, 
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2, 
offset=1892533025; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
 in thread 
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State 
transition from RUNNING to PENDING_SHUTDOWN
{code}

What is wrong there?


> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:

[jira] [Commented] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612374#comment-16612374
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7214:
---

[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when 
they need to handle the case below?

{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled 
exception: Exception caught in process. taskId=0_2, 
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2, 
offset=1892533025; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
 
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
 
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
 in thread 
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
 Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client 
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State 
transition from RUNNING to PENDING_SHUTDOWN
{code}

What is wrong there?


> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612356#comment-16612356
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6699:
---

[~mjsax]I understand you point, but still I do not see why we need more than 2 
kafka brokers. Especially that we have 5 separate zookeeper nodes.

> When one of two Kafka nodes are dead, streaming API cannot handle messaging
> ---
>
> Key: KAFKA-6699
> URL: https://issues.apache.org/jira/browse/KAFKA-6699
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> I am observing quite often, when Kafka Broker is partly dead(*), then 
> application, which uses streaming API are doing nothing.
> (*) Partly dead in my case it means that one of two Kafka nodes are out of 
> order. 
> Especially when disk is full on one machine, then Broker is going in some 
> strange state, where streaming API goes vacations. It seems like regular 
> producer/consumer API has no problem in such a case.
> Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606846#comment-16606846
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7214:
---

Hi [~vvcephei],

This is what I had stayed in other problem reports.
Memory consumption or memory model described in Kafka documentation does not 
fit to reality.

Now I am in the phase, when I am fully guessing, by obtaining mystic fatals 
during data processing.
Is it expected, to increase memory when I get any kind of error in Kafka?

I would be really greatful when I can more less, even with 30% overhead, but 
calculate, how much memory I need for my service to process X Msg/s with given 
size and given retention and given whatever. But it is not the case.

And again. If error reporting would be -> Out Of Memory. I would also quickly 
see that is is really memory issue and I would be able to calculate this 
myself. But ending with:

{code}
2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close
task due to the following error:
org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0
{code}

Does not even point to the problem, and still I am not sure if it is really 
memory problem, I see only, that when I give more memory, I do not see it that 
often.

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-7363) How num.stream.threads in streaming application influence memory consumption?

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606830#comment-16606830
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7363:
---

I would love to do that, but currently in in the epoch of shotgun debugging, 
then guessing and impact configuring. This is not the state I can contribute 
anything :-(.

> How num.stream.threads in streaming application influence memory consumption?
> -
>
> Key: KAFKA-7363
> URL: https://issues.apache.org/jira/browse/KAFKA-7363
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> How option _num.stream.threads_ in streaming application influence memory 
> consumption?
> I see that by increasing num.stream.threads my application needs more memory.
> This is obvious, but it is not obvious how much I need to give it. Try and 
> error method does not work, as it seems to be highly dependen on forced 
> throughput.
> I mean: higher load more memory is needed.
> Thanks for help and regards,
> Seweryn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606816#comment-16606816
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6777 at 9/7/18 8:00 AM:
---

What means long pauses? 
I see Broker is not doing anything for hours. 
Is it expected behaviour of CG?


was (Author: habdank):
What means long pauses? 
I see Broker is not doing anythink for hours. 
Is it expected behaviour of CG?

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606816#comment-16606816
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6777:
---

What means long pauses? 
I see Broker is not doing anythink for hours. 
Is it expected behaviour of CG?

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606813#comment-16606813
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6699 at 9/7/18 7:57 AM:
---

OK. Misunderstanding. 

We have replication factor 2. And in topic _describe _I see Isr: 1,2. I am 
still sure, we do not have wrong configuration.

And if, we would have, such a simple check like consistency between replication 
factor and number of brokers, shall be as warning in client startup reported.


was (Author: habdank):
OK. Misunderstanding. 

We have replication factor 2. And in topic describe I see Isr: 1,2. I am still 
sure, we do not have wrong configuration.

And if, we would have, such a simple check like consistency between replication 
factor and number of brokers, shall be as warning in client startup reported.

> When one of two Kafka nodes are dead, streaming API cannot handle messaging
> ---
>
> Key: KAFKA-6699
> URL: https://issues.apache.org/jira/browse/KAFKA-6699
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> I am observing quite often, when Kafka Broker is partly dead(*), then 
> application, which uses streaming API are doing nothing.
> (*) Partly dead in my case it means that one of two Kafka nodes are out of 
> order. 
> Especially when disk is full on one machine, then Broker is going in some 
> strange state, where streaming API goes vacations. It seems like regular 
> producer/consumer API has no problem in such a case.
> Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KAFKA-6457) Error: NOT_LEADER_FOR_PARTITION leads to NPE

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-6457.
-

> Error: NOT_LEADER_FOR_PARTITION leads to NPE
> 
>
> Key: KAFKA-6457
> URL: https://issues.apache.org/jira/browse/KAFKA-6457
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
> Fix For: 1.0.1
>
>
> One of our nodes was dead. Then the second one tooks all responsibility.
> But streamming aplication in the meanwhile crashed due to NPE caused by 
> {{Error: NOT_LEADER_FOR_PARTITION}}.
> The stack trace is below.
>  
> Is it something expected?
>  
> {code:java}
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer ...2018-01-17 
> 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-5, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-7, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [ERROR] AbstractCoordinator:296 - [Consumer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-consumer,
>  groupId=restreamer-my] Heartbeat thread for group restreamer-my failed due 
> to unexpected error
> java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436) 
> ~[my-restreamer.jar:?]
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:395) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934)
>  [my-restreamer.jar:?]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-5882.
-

Not more visible in Kafka 1.1.1

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
> Fix For: 1.1.1
>
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605883#comment-16605883
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

In kafka 1.1.1 I do not see this anymore.
Shall I close the issue with comment about that?

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KAFKA-6457) Error: NOT_LEADER_FOR_PARTITION leads to NPE

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-6457.
-

In Kafka 1.1.1 see it no more.

> Error: NOT_LEADER_FOR_PARTITION leads to NPE
> 
>
> Key: KAFKA-6457
> URL: https://issues.apache.org/jira/browse/KAFKA-6457
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
> Fix For: 1.0.1
>
>
> One of our nodes was dead. Then the second one tooks all responsibility.
> But streamming aplication in the meanwhile crashed due to NPE caused by 
> {{Error: NOT_LEADER_FOR_PARTITION}}.
> The stack trace is below.
>  
> Is it something expected?
>  
> {code:java}
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer ...2018-01-17 
> 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-5, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-7, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [ERROR] AbstractCoordinator:296 - [Consumer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-consumer,
>  groupId=restreamer-my] Heartbeat thread for group restreamer-my failed due 
> to unexpected error
> java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436) 
> ~[my-restreamer.jar:?]
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:395) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934)
>  [my-restreamer.jar:?]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605867#comment-16605867
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6699:
---

ISR: 2
ack: 1

Do you suggest to have more brokers?

> When one of two Kafka nodes are dead, streaming API cannot handle messaging
> ---
>
> Key: KAFKA-6699
> URL: https://issues.apache.org/jira/browse/KAFKA-6699
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> I am observing quite often, when Kafka Broker is partly dead(*), then 
> application, which uses streaming API are doing nothing.
> (*) Partly dead in my case it means that one of two Kafka nodes are out of 
> order. 
> Especially when disk is full on one machine, then Broker is going in some 
> strange state, where streaming API goes vacations. It seems like regular 
> producer/consumer API has no problem in such a case.
> Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605851#comment-16605851
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/6/18 2:40 PM:
---

Problem is that  KSTREAM-SOURCE-X is mostly  KSTREAM-SOURCE-0 
independently of which process and how much processes are running (or trying to 
run).

How I reproduce error at my side. Let's assume I have low message flow < 100 
Msg/sec Msg size ~ 1kB.

I am starting app using streaming API. This app reads from 30 topics and send 
messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.

At second server I am starting. second instance. The same. It is starting.

The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec 
Msg size ~ 1kB.

I am starting app using streaming API. This app reads from 30 topics and send 
messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even if memory spec 
stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB 
leads at some point to yet another mystic crasches.

At second server I am starting. second instance. If I am starting it with 300 
MB - I got immediately this error. Application tries to starrt, but then I got 
this error and all affected topics are goig to be dead. If I am giving 1GB, it 
is better application works some hours, but any minimal peak aroud 5000 Msg/s 
to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes 
with 5GB. they could work continuously like 2-4 days.

I am sorry I have no better description.
Once I tried to start TRACE level logs in Kafka, but this is impossible with 
message flow at 5000 Msg/s.




was (Author: habdank):
Problem is that  KSTREAM-SOURCE-X is mostly  KSTREAM-SOURCE-0 
independently of which process and how much processes are running (or trying to 
run).

How I reproduce error at my side. Let's assume I have low message flow < 100 
Msg/sec Msg size ~ 1kB.

I am starting app using streaming API. This app reads from 30 topics and send 
messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.

At second server I am starting. second instance. The same. It is starting.

The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec 
Msg size ~ 1kB.

I am starting app using streaming API. This app reads from 30 topics and send 
messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even in memory spec 
stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB 
leads at some point to yet another mystic crasches.

At second server I am starting. second instance. If I am starting it with 300 
MB - I got immediately this error. Application tries to starrt, but then I got 
this error and all affected topics are goig to be dead. If I am giving 1GB, it 
is better application works some hours, but any minimal peak aroud 5000 Msg/s 
to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes 
with 5GB. they could work continuously like 2-4 days.

I am sorry I have no better description.
Once I tried to start TRACE level logs in Kafka, but this is impossible with 
message flow at 5000 Msg/s.



> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> 

[jira] [Commented] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605851#comment-16605851
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7214:
---

Problem is that  KSTREAM-SOURCE-X is mostly  KSTREAM-SOURCE-0 
independently of which process and how much processes are running (or trying to 
run).

How I reproduce error at my side. Let's assume I have low message flow < 100 
Msg/sec Msg size ~ 1kB.

I am starting app using streaming API. This app reads from 30 topics and send 
messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.

At second server I am starting. second instance. The same. It is starting.

The other case. Let's assume I have low message flow > 5000 Msg/sec Msg size ~ 
1kB.

I am starting app using streaming API. This app reads from 30 topics and send 
messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even in memory spec 
stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB 
leads at some point to yet another mystic crasches.

At second server I am starting. second instance. If I am starting it with 300 
MB - I got immediately this error. Application tries to starrt, but then I got 
this error and all affected topics are goig to be dead. If I am giving 1GB, it 
is better application works some hours, but any minimal peak aroud 5000 Msg/s 
to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes 
with 5GB. they could work continuously like 2-4 days.

I am sorry I have no better description.
Once I tried to start TRACE level logs in Kafka, but this is impossible with 
message flow at 5000 Msg/s.



> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KAFKA-7363) How num.stream.threads in streaming application influence memory consumption?

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-7363.
-

> How num.stream.threads in streaming application influence memory consumption?
> -
>
> Key: KAFKA-7363
> URL: https://issues.apache.org/jira/browse/KAFKA-7363
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> How option _num.stream.threads_ in streaming application influence memory 
> consumption?
> I see that by increasing num.stream.threads my application needs more memory.
> This is obvious, but it is not obvious how much I need to give it. Try and 
> error method does not work, as it seems to be highly dependen on forced 
> throughput.
> I mean: higher load more memory is needed.
> Thanks for help and regards,
> Seweryn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-7363) How num.stream.threads in streaming application influence memory consumption?

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603250#comment-16603250
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7363:
---

Thanks for hints.

Honestly spoken I would expect extended information in the documentation, 
as it is now completely impossible to plan server resources when using kafka 
streams.

I was experimenting quite a lot, and I cannot clearly create my resource model.

I mean I am getting from operation and customers information how many messages 
per second they will be send and either how many MB perseconds they will send 
or what is the avarage message size.

Then I need to allocate more resources - clear. But how much?

I was guessing, and there give more memory or CPUs. 
Cool, but this is not really engineering, this is speculation.
When comes next users with yet another requirements, I will guess again - not 
very effective.

But I am accepting proposal to send mail to the mailing list and closing ticket.

> How num.stream.threads in streaming application influence memory consumption?
> -
>
> Key: KAFKA-7363
> URL: https://issues.apache.org/jira/browse/KAFKA-7363
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> How option _num.stream.threads_ in streaming application influence memory 
> consumption?
> I see that by increasing num.stream.threads my application needs more memory.
> This is obvious, but it is not obvious how much I need to give it. Try and 
> error method does not work, as it seems to be highly dependen on forced 
> throughput.
> I mean: higher load more memory is needed.
> Thanks for help and regards,
> Seweryn.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7375) Improve error messages verbosity

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-7375:
-

 Summary: Improve error messages verbosity
 Key: KAFKA-7375
 URL: https://issues.apache.org/jira/browse/KAFKA-7375
 Project: Kafka
  Issue Type: Task
Affects Versions: 1.1.1
Reporter: Seweryn Habdank-Wojewodzki


Dears,

Very often when clients are trying to connect we see in Kafka logs:

{code}
“org.apache.kafka.common.network.SslTransportLayer  - Failed to send SSL Close 
message“
{code}

The problem here is following: there is no word who? No IP, no client addres, 
nothing.

Would be great to have in all error or warning reports like this one, very 
precize information which client has a problem, to be able to solve it. When 
the number of clients is more than 10, this message is completely useless and 
when there are even more clients it really spams logs.

Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598305#comment-16598305
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 8/31/18 6:27 AM:


The keyword in all those errors is: KSTREAM-SOURCE-X

Kafka 1.1.1 makes it even more horrible :-(, but ... 

... it seems this is related to memory consumption and perhaps number of 
threads used by streaming app.
I had increased JVM heap from 348 MB to 1GB and decreased number of threads 
from 16 to 2 and it seems not happened so often.

I will check this further.

But I am going back to my comment from bug report KAFKA-6777. I think (after 
code review), there are very many places in code, where potentially OutOfMemory 
error ist not handled properly and they could be converted in any kind of 
random errors or even completely swallowed giving random behaviour of clients 
or servers.

I would expect, that OutOfMemory will lead to fast application crash with clear 
infomation, where is the problem.


was (Author: habdank):
Kafka 1.1.1 makes it even more horrible :-(, but ... 

... it seems this is related to memory consumption and perhaps number of 
threads used by streaming app.
I had increased JVM heap from 348 MB to 1GB and decreased number of threads 
from 16 to 2 and it seems not happened so often.

I will check this further.

But I am going back to my comment from bug report KAFKA-6777. I think (after 
code review), there are very many places in code, where potentially OutOfMemory 
error ist not handled properly and they could be converted in any kind of 
random errors or even completely swallowed giving random behaviour of clients 
or servers.

I would expect, that OutOfMemory will lead to fast application crash with clear 
infomation, where is the problem.

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598305#comment-16598305
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7214:
---

Kafka 1.1.1 makes it even more horrible :-(, but ... 

... it seems this is related to memory consumption and perhaps number of 
threads used by streaming app.
I had increased JVM heap from 348 MB to 1GB and decreased number of threads 
from 16 to 2 and it seems not happened so often.

I will check this further.

But I am going back to my comment from bug report KAFKA-6777. I think (after 
code review), there are very many places in code, where potentially OutOfMemory 
error ist not handled properly and they could be converted in any kind of 
random errors or even completely swallowed giving random behaviour of clients 
or servers.

I would expect, that OutOfMemory will lead to fast application crash with clear 
infomation, where is the problem.

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KAFKA-6459) By (Re-)joining group StreamThread got NPE

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-6459.
-

Verified with Kafka 1.1.0.

> By (Re-)joining group StreamThread got NPE
> --
>
> Key: KAFKA-6459
> URL: https://issues.apache.org/jira/browse/KAFKA-6459
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
> Fix For: 1.0.1
>
>
> We encouteres more instabilities in Kafka Streams.
> By (Re-)joining group StreamThread got NPE.
> {code}
> 2018-01-18 09:48:44 INFO  AbstractCoordinator:336 - [Consumer 
> clientId=kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1-consumer,
>  groupId=kafka-endpoint] (Re-)joining group
> 2018-01-18 09:48:44 INFO  StreamPartitionAssignor:341 - stream-thread 
> [kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1-consumer] 
> Assigned tasks to clients as 
> \{c1e67c77-c0c8-413d-b0f2-31f22bdeae05=[activeTasks: ([0_0, 0_1, 0_2, 0_3, 
> 0_4, 0_5, 0_6, 0_7, 0_8, 0_9]) standbyTasks: ([]) assignedTasks: ([0_0, 0_1, 
> 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8, 0_9]) prevActiveTasks: ([0_0, 0_1, 0_2, 
> 0_3, 0_4]) prevAssignedTasks: ([0_0, 0_1, 0_2, 0_3, 0_4]) capacity: 1]}.
> 2018-01-18 09:48:44 INFO  AbstractCoordinator:341 - [Consumer 
> clientId=kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Successfully joined group with generation 3950
> 2018-01-18 09:48:44 INFO  ConsumerCoordinator:341 - [Consumer 
> clientId=kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Setting newly assigned partitions 
> [my_internal_topic-6, my_internal_topic-5, my_internal_topic-8, 
> my_internal_topic-7, my_internal_topic-9, my_internal_topic-0, 
> my_internal_topic-2, my_internal_topic-1, my_internal_topic-4, 
> my_internal_topic-3]
> 2018-01-18 09:48:44 INFO  StreamThread:346 - stream-thread 
> [kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1] State 
> transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED
> 2018-01-18 09:48:44 INFO  StreamThread:351 - stream-thread 
> [kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1] 
> partition assignment took 149 ms.
>     current active tasks: [0_0, 0_1, 0_2, 0_3, 0_4, 0_5, 0_6, 0_7, 0_8, 
> 0_9]
>     current standby tasks: []
>     previous active tasks: [0_0, 0_1, 0_2, 0_3, 0_4]
> 2018-01-18 09:48:44 ERROR StreamThread:306 - stream-thread 
> [kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1] 
> Encountered the following error during processing:
> java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436) 
> ~[my-kafka-endpoint.jar:?]
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:395) 
> ~[my-kafka-endpoint.jar:?]
>     at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) 
> ~[my-kafka-endpoint.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238)
>  ~[my-kafka-endpoint.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275)
>  ~[my-kafka-endpoint.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934)
>  ~[my-kafka-endpoint.jar:?]
> 2018-01-18 09:48:44 INFO  StreamThread:346 - stream-thread 
> [kafka-endpoint-c1e67c77-c0c8-413d-b0f2-31f22bdeae05-StreamThread-1] State 
> transition from PARTITIONS_ASSIGNED to PENDING_SHUTDOWN
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587475#comment-16587475
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-7214:
---

Hi,

I had updated Kafka client to 1.1.1. I have similar.

max.poll.interval.ms = 10 000 000 ~ 2,7 Hours
max.poll.records=500

Usual system message system processing is ~ 5000 Msg/s

2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close
task due to the following error:
org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0
ProcessorTopology:
KSTREAM-SOURCE-00:
topics: [my_topic]
children:   [KSTREAM-FILTER-01]
KSTREAM-FILTER-01:
children:   [KSTREAM-MAP-02]
KSTREAM-MAP-02:
children:   [KSTREAM-SINK-03]
KSTREAM-SINK-03:
topic:  other_topic
Partitions [my_topic-0]

at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets
(StreamTask.java:380) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St
reamTask.java:53) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT
ask.java:316) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu
reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream
Task.java:307) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea
mTask.java:440) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT
ask.java:546) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi
gnedTasks.java:405) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.TaskManager.shutdown(Tas
kManager.java:260) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.completeShu
tdown(StreamThread.java:) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamT
hread.java:730) [restreamer.jar:?]
Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Commit canno
t be completed since the group has already rebalanced and assigned the partition
s to another member. This means that the time between subsequent calls to poll()
 was longer than the configured max.poll.interval.ms, which typically implies th
at the poll loop is spending too much time message processing. You can address t
his either by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-7214:
--
Affects Version/s: 1.1.1

> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=my_instance_medium_topic, partition=1, offset=198900203; 
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>  
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>  
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
>  in thread 
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
>  State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587475#comment-16587475
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 8/21/18 2:11 PM:


Hi,

I had updated Kafka client to 1.1.1. I have similar situation.

max.poll.interval.ms = 10 000 000 ~ 2,7 Hours
max.poll.records=500

Usual system message system processing is ~ 5000 Msg/s

2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close
task due to the following error:
org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0
ProcessorTopology:
KSTREAM-SOURCE-00:
topics: [my_topic]
children:   [KSTREAM-FILTER-01]
KSTREAM-FILTER-01:
children:   [KSTREAM-MAP-02]
KSTREAM-MAP-02:
children:   [KSTREAM-SINK-03]
KSTREAM-SINK-03:
topic:  other_topic
Partitions [my_topic-0]

at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets
(StreamTask.java:380) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St
reamTask.java:53) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT
ask.java:316) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu
reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream
Task.java:307) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea
mTask.java:440) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT
ask.java:546) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi
gnedTasks.java:405) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.TaskManager.shutdown(Tas
kManager.java:260) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.completeShu
tdown(StreamThread.java:) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamT
hread.java:730) [restreamer.jar:?]
Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Commit canno
t be completed since the group has already rebalanced and assigned the partition
s to another member. This means that the time between subsequent calls to poll()
 was longer than the configured max.poll.interval.ms, which typically implies th
at the poll loop is spending too much time message processing. You can address t
his either by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.

Regards,
Seweryn.


was (Author: habdank):
Hi,

I had updated Kafka client to 1.1.1. I have similar.

max.poll.interval.ms = 10 000 000 ~ 2,7 Hours
max.poll.records=500

Usual system message system processing is ~ 5000 Msg/s

2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close
task due to the following error:
org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0
ProcessorTopology:
KSTREAM-SOURCE-00:
topics: [my_topic]
children:   [KSTREAM-FILTER-01]
KSTREAM-FILTER-01:
children:   [KSTREAM-MAP-02]
KSTREAM-MAP-02:
children:   [KSTREAM-SINK-03]
KSTREAM-SINK-03:
topic:  other_topic
Partitions [my_topic-0]

at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets
(StreamTask.java:380) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St
reamTask.java:53) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT
ask.java:316) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu
reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream
Task.java:307) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea
mTask.java:440) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT
ask.java:546) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi
gnedTasks.java:405) [restreamer.jar:?]
at 

[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587475#comment-16587475
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 8/21/18 2:10 PM:


Hi,

I had updated Kafka client to 1.1.1. I have similar.

max.poll.interval.ms = 10 000 000 ~ 2,7 Hours
max.poll.records=500

Usual system message system processing is ~ 5000 Msg/s

2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close
task due to the following error:
org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0
ProcessorTopology:
KSTREAM-SOURCE-00:
topics: [my_topic]
children:   [KSTREAM-FILTER-01]
KSTREAM-FILTER-01:
children:   [KSTREAM-MAP-02]
KSTREAM-MAP-02:
children:   [KSTREAM-SINK-03]
KSTREAM-SINK-03:
topic:  other_topic
Partitions [my_topic-0]

at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets
(StreamTask.java:380) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St
reamTask.java:53) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT
ask.java:316) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu
reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream
Task.java:307) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea
mTask.java:440) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT
ask.java:546) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi
gnedTasks.java:405) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.TaskManager.shutdown(Tas
kManager.java:260) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.completeShu
tdown(StreamThread.java:) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamT
hread.java:730) [restreamer.jar:?]
Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Commit canno
t be completed since the group has already rebalanced and assigned the partition
s to another member. This means that the time between subsequent calls to poll()
 was longer than the configured max.poll.interval.ms, which typically implies th
at the poll loop is spending too much time message processing. You can address t
his either by increasing the session timeout or by reducing the maximum size of
batches returned in poll() with max.poll.records.

Regards,
Seweryn.


was (Author: habdank):
Hi,

I had updated Kafka client to 1.1.1. I have similar.

max.poll.interval.ms = 10 000 000 ~ 2,7 Hours
max.poll.records=500

Usual system message system processing is ~ 5000 Msg/s

2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close
task due to the following error:
org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0
ProcessorTopology:
KSTREAM-SOURCE-00:
topics: [my_topic]
children:   [KSTREAM-FILTER-01]
KSTREAM-FILTER-01:
children:   [KSTREAM-MAP-02]
KSTREAM-MAP-02:
children:   [KSTREAM-SINK-03]
KSTREAM-SINK-03:
topic:  other_topic
Partitions [my_topic-0]

at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets
(StreamTask.java:380) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St
reamTask.java:53) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT
ask.java:316) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu
reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream
Task.java:307) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea
mTask.java:440) ~[restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT
ask.java:546) [restreamer.jar:?]
at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi
gnedTasks.java:405) [restreamer.jar:?]
at 

[jira] [Created] (KAFKA-7214) Mystic FATAL error

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-7214:
-

 Summary: Mystic FATAL error
 Key: KAFKA-7214
 URL: https://issues.apache.org/jira/browse/KAFKA-7214
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.3
Reporter: Seweryn Habdank-Wojewodzki


Dears,

Very often at startup of the streaming application I got exception:

{code}
Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
topic=my_instance_medium_topic, partition=1, offset=198900203; 
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
 
org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
 
org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
 
org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
 
org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
 
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
 in thread 
my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
{code}

and then (without shutdown request from my side):

{code}
2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread 
[my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
 State transition from PENDING_SHUTDOWN to DEAD.
{code}

What is this?
How to correctly handle it?

Thanks in advance for help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-6882) Wrong producer settings may lead to DoS on Kafka Server

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-6882:
--
Description: 
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading, that for safety, 
they need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safety matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some logic to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?


  was:
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading, that for safety, 
they need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safety matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?



> Wrong 

[jira] [Updated] (KAFKA-6882) Wrong producer settings may lead to DoS on Kafka Server

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-6882:
--
Description: 
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading, that for safety, 
they need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safety matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?


  was:
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading, that for safety, 
they need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safetly matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?



> Wrong 

[jira] [Updated] (KAFKA-6882) Wrong producer settings may lead to DoS on Kafka Server

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-6882:
--
Description: 
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading, that for safety, 
they need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safetly matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?


  was:
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading that for safety they 
need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safetly matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?



> Wrong 

[jira] [Updated] (KAFKA-6882) Wrong producer settings may lead to DoS on Kafka Server

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-6882:
--
Description: 
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading that for safety they 
need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safetly matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?


  was:
The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers they are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading that for safety they 
need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safetly matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?



> 

[jira] [Created] (KAFKA-6882) Wrong producer settings may lead to DoS on Kafka Server

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-6882:
-

 Summary: Wrong producer settings may lead to DoS on Kafka Server
 Key: KAFKA-6882
 URL: https://issues.apache.org/jira/browse/KAFKA-6882
 Project: Kafka
  Issue Type: Bug
  Components: core, producer 
Affects Versions: 1.0.1, 1.1.0
Reporter: Seweryn Habdank-Wojewodzki


The documentation of the following parameters “linger.ms” and “batch.size” is a 
bit confusing. In fact those parameters wrongly set on the producer side might 
completely destroy BROKER throughput.

I see, that smart developers they are reading documentation of those parameters.
Then they want to have super performance and super safety, so they set 
something like this below:

{code}
kafkaProps.put(ProducerConfig.LINGER_MS_CONFIG, 1);
kafkaProps.put(ProducerConfig.BATCH_SIZE_CONFIG, 0);
{code}

Then we have situation, when each and every message is send separately. TCP/IP 
protocol is really busy in that case and when they needed high throughput they 
got much less throughput, as every message is goes separately causing all 
network communication and TCP/IP overhead significant.

Those settings are good only if someone sends critical messages like once a 
while (e.g. one message per minute) and not when throughput is important by 
sending thousands messages per second.

Situation is even worse when smart developers are reading that for safety they 
need acknowledges from all cluster nodes. So they are adding:

{code}
kafkaProps.put(ProducerConfig.ACKS_CONFIG, "all");
{code}

And this is the end of Kafka performance! 

Even worse it is not a problem for the Kafka producer. The problem remains at 
the server (cluster, broker) side. The server is so busy by acknowledging *each 
and every* message from all nodes, that other work is NOT performed, so the end 
to end performance is almost none.

I would like to ask you to improve documentation of this parameters.
And consider corner cases is case of providing detailed information how extreme 
values of parameters - namely lowest and highest – may influence work of the 
cluster.
This was documentation issue. 

On the other hand it is security/safetly matter.

Technically the problem is that __commit_offsets topic is loaded with enormous 
amount of messages. It leads to the situation, when Kafka Broker is exposed to 
*DoS *due to the Producer settings. Three lines of code a bit load and the 
Kafka cluster is dead.
I suppose there are ways to prevent such a situation on the cluster side, but 
it require some loginc to be implemented to detect such a simple but efficient 
DoS.

BTW. Do Kafka Admin Tools provide any kind of "kill" connection, when one or 
the other producer makes problems?




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437181#comment-16437181
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6777:
---

Our options:
-server
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC
-Djava.awt.headless=true

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433863#comment-16433863
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6777 at 4/11/18 12:58 PM:
-

One more comment. I see quite often in Kafka that Throwable is converted to 
RuntimeException.
 This kind of code may lead to situation when OOM will never appear.

I had made simple example:
{code:java}
public class Main {
public static void main ( String[] args ) {
try {
try {
throw new OutOfMemoryError();
// very often in Kafka code:
} catch ( Throwable t ) {
throw ( RuntimeException ) t;
}
// end of very often
} catch ( Exception ignore ) {
}
}
}
{code}
Executed with:
{code:java}
-XX:OnOutOfMemoryError="echo OOM"
{code}
leads to:
{code:java}
Process finished with exit code 0
{code}
I see no *OOM* string, also no _OutOfMemoryError_ is noticed, by any stactrace.

Generally all Errors derived from {{java.lang.Error}} are swallowed, including:
* InternalError, 
* OutOfMemoryError, 
* StackOverflowError, 
* UnknownError, 
* ThreadDeath,
* IOError
 


was (Author: habdank):
One more comment. I see quite often in Kafka that Throwable is converted to 
RuntimeException.
 This kind of code may lead to situation when OOM will never appear.

I had made simple example:
{code:java}
public class Main {
public static void main ( String[] args ) {
try {
try {
throw new OutOfMemoryError();
// very often in Kafka code:
} catch ( Throwable t ) {
throw ( RuntimeException ) t;
}
// end of very often
} catch ( Exception ignore ) {
}
}
}
{code}
Executed with:
{code:java}
-XX:OnOutOfMemoryError="echo OOM"
{code}
leads to:
{code:java}
Process finished with exit code 0
{code}
I see no *OOM* string, also no _OutOfMemoryError_ is noticed, by any stactrace.

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433863#comment-16433863
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6777:
---

One more comment. I see quite often in Kafka that Throwable is converted to 
RuntimeException.
 This kind of code may lead to situation when OOM will never appear.

I had made simple example:
{code:java}
public class Main {
public static void main ( String[] args ) {
try {
try {
throw new OutOfMemoryError();
// very often in Kafka code:
} catch ( Throwable t ) {
throw ( RuntimeException ) t;
}
// end of very often
} catch ( Exception ignore ) {
}
}
}
{code}
Executed with:
{code:java}
-XX:OnOutOfMemoryError="echo OOM"
{code}
leads to:
{code:java}
Process finished with exit code 0
{code}
I see no *OOM* string, also no _OutOfMemoryError_ is noticed, by any stactrace.

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433799#comment-16433799
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6777 at 4/11/18 12:17 PM:
-

Thanks for comment.

The problem is, that either the OnOutOfMemoryError is never thrown, as the 
algorithms are trying to do their best and they are loading GC, so later no 
message processing may happen, as most CPU is used by GC.

Or the OnOutOfMemoryError is thrown, but caught in code like catch(Throwable) {}

The observed bahaviour is that at INFO level logs there is no explicit error 
like: OnOutOfMemoryError. 
I had seen in JMX metrics and there heap is out and GC is endless busy, till 
nothing is also to JMX reported.

I mean I can write a tool to reboot Kafka node, when GC load on CPU is higher 
than 40% or so, but this kind of tool is workaround and not a solution for the 
problem.

I am attaching graphs to highlight wat had happened.

On the image blow there are metrics from 2 Kafka nodes. The green one was 
dead/zombie when GC time reached 80%. This "drop" of value is only a 
presentation matter.

!screenshot-1.png! 


was (Author: habdank):
Thanks for comment.

The problem is, that either the OnOutOfMemoryError is never thrown, as the 
algorithms trying to do their best and they are loading GC, and then no message 
processing may happen.

Or the OnOutOfMemoryError is thrown, but caught in code like catch(Throwable) {}

The observed bahaviour is that at INFO level logs there is no explicit error 
like: OnOutOfMemoryError. 
I had seen in JMX metrics and there heap is out and GC is endless busy, till 
nothing is also to JMX reported.

I mean I can write a tool to reboot Kafka node, when GC load on CPU is higher 
than 40% or so, but this kind of tool is workaround and not a solution for the 
problem.

I am attaching graphs to highlight wat had happend.

On the image blow there are metrics from 2 Kafka nodes. The green one was 
dead/zombie when GC time reached 80%. This "drop" of value is only a 
presentation matter.

 !screenshot-1.png! 

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433799#comment-16433799
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6777 at 4/11/18 12:14 PM:
-

Thanks for comment.

The problem is, that either the OnOutOfMemoryError is never thrown, as the 
algorithms trying to do their best and they are loading GC, and then no message 
processing may happen.

Or the OnOutOfMemoryError is thrown, but caught in code like catch(Throwable) {}

The observed bahaviour is that at INFO level logs there is no explicit error 
like: OnOutOfMemoryError. 
I had seen in JMX metrics and there heap is out and GC is endless busy, till 
nothing is also to JMX reported.

I mean I can write a tool to reboot Kafka node, when GC load on CPU is higher 
than 40% or so, but this kind of tool is workaround and not a solution for the 
problem.

I am attaching graphs to highlight wat had happend.

On the image blow there are metrics from 2 Kafka nodes. The green one was 
dead/zombie when GC time reached 80%. This "drop" of value is only a 
presentation matter.

 !screenshot-1.png! 


was (Author: habdank):
Thanks for comment.

The problem is, that either the OnOutOfMemoryError is never thrown, as the 
algorithms trying to do their best and they are loading GC, and then no message 
processing may happen.

Or the OnOutOfMemoryError is thrown, but caught in code like catch(Throwable) {}

The observed bahaviour is that at INFO level logs there is no explicit error 
like: OnOutOfMemoryError. 
I had seen in JMX metrics and there heap is out and GC is endless busy, till 
nothing is also to JMX reported.

I mean I can write a tool to reboot Kafka node, when GC load on CPU is higher 
than 40% or so, but this kind of tool is workaround and not a solution for the 
problem.

I am attaching graphs to highlight wat had happend.

 !screenshot-1.png! 

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433799#comment-16433799
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6777:
---

Thanks for comment.

The problem is, that either the OnOutOfMemoryError is never thrown, as the 
algorithms trying to do their best and they are loading GC, and then no message 
processing may happen.

Or the OnOutOfMemoryError is thrown, but caught in code like catch(Throwable) {}

The observed bahaviour is that at INFO level logs there is no explicit error 
like: OnOutOfMemoryError. 
I had seen in JMX metrics and there heap is out and GC is endless busy, till 
nothing is also to JMX reported.

I mean I can write a tool to reboot Kafka node, when GC load on CPU is higher 
than 40% or so, but this kind of tool is workaround and not a solution for the 
problem.

I am attaching graphs to highlight wat had happend.

 !screenshot-1.png! 

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-6777:
--
Attachment: screenshot-1.png

> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
> Attachments: screenshot-1.png
>
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-6777:
--
Description: 
Dears,

We already encountered many times problems related to Out Of Memory situation 
in Kafka Broker and streaming clients.

The scenario is the following.

When Kafka Broker (or Streaming Client) is under load and has too less memory, 
there are no errors in server logs. One can see some cryptic entries in GC 
logs, but they are definitely not self-explaining.

Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
monitoring, that JVM uses more and more time in GC. In our case it grows from 
e.g. 1% to 80%-90% of CPU time is used by GC.

Next, software collapses into zombie mode – process in not ending. In such a 
case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
Kafka treats such a zombie process normal and somewhat sends messages, which 
are in fact getting lost, also the cluster is not excluding broken nodes. The 
question is how to configure Kafka to really terminate the JVM and not remain 
in zombie mode, to give a chance to other nodes to know, that something is dead.

I would expect that in Out Of Memory situation JVM is ended if not graceful 
than at least process is crashed.

  was:
Dears,

We already encountered many times problems related to Out Of Memory situation 
in Kafka Broker and streaming clients.

The scenario is the following.

When Kafka Broker (or Streaming Client) is under load and has too less memory, 
there are no errors in server logs. One can see some cryptic entries in GC 
logs, but they are definitely not self-explaining.

Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
monitoring, that JVM uses more and more time in GC. In our case it grows from 
e.g. 1% to 80%-90% of CPU time is used by GC.

Next software collapses into zombie mode – process in not ending. In such a 
case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
Kafka treats such a zombie process normal and somewhat sends messages, which 
are in fact getting lost, also the cluster is not excluding broken nodes. The 
question is how to configure Kafka to really terminate the JVM and not remain 
in zombie mode, to give a chance to other nodes to know, that something is dead.

I would expect that in Out Of Memory situation JVM is ended if not graceful 
than at least process is crashed.


> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next, software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-6777:
--
Description: 
Dears,

We already encountered many times problems related to Out Of Memory situation 
in Kafka Broker and streaming clients.

The scenario is the following.

When Kafka Broker (or Streaming Client) is under load and has too less memory, 
there are no errors in server logs. One can see some cryptic entries in GC 
logs, but they are definitely not self-explaining.

Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
monitoring, that JVM uses more and more time in GC. In our case it grows from 
e.g. 1% to 80%-90% of CPU time is used by GC.

Next software collapses into zombie mode – process in not ending. In such a 
case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
Kafka treats such a zombie process normal and somewhat sends messages, which 
are in fact getting lost, also the cluster is not excluding broken nodes. The 
question is how to configure Kafka to really terminate the JVM and not remain 
in zombie mode, to give a chance to other nodes to know, that something is dead.

I would expect that in Out Of Memory situation JVM is ended if not graceful 
than at least process is crashed.

  was:
Dears,

We already encountered many times problems related to Out Of Memory situation 
in Kafka Broker and streaming clients.

The scenario is the following.

When Kafka Broker (or Streaming Client) is under load and has too less memory, 
there are no errors in server logs. One can see some cryptic entries in GC 
logs, but they are definitely not self-explaining.

Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
monitoring, that JVM uses more and more time in GC. In our case it grows from 
e. 1% to 80%-90% of CPU time is used by GC.

Next software collapses into zombie mode – process in not ending. In such a 
case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
Kafka treats such a zombie process normal and somewhat sends messages, which 
are in fact getting lost, also the cluster is not excluding broken nodes. The 
question is how to configure Kafka to really terminate the JVM and not remain 
in zombie mode, to give a chance to other nodes to know, that something is dead.

I would expect that in Out Of Memory situation JVM is ended if not graceful 
than at least process is crashed.


> Wrong reaction on Out Of Memory situation
> -
>
> Key: KAFKA-6777
> URL: https://issues.apache.org/jira/browse/KAFKA-6777
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> We already encountered many times problems related to Out Of Memory situation 
> in Kafka Broker and streaming clients.
> The scenario is the following.
> When Kafka Broker (or Streaming Client) is under load and has too less 
> memory, there are no errors in server logs. One can see some cryptic entries 
> in GC logs, but they are definitely not self-explaining.
> Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
> monitoring, that JVM uses more and more time in GC. In our case it grows from 
> e.g. 1% to 80%-90% of CPU time is used by GC.
> Next software collapses into zombie mode – process in not ending. In such a 
> case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
> Kafka treats such a zombie process normal and somewhat sends messages, which 
> are in fact getting lost, also the cluster is not excluding broken nodes. The 
> question is how to configure Kafka to really terminate the JVM and not remain 
> in zombie mode, to give a chance to other nodes to know, that something is 
> dead.
> I would expect that in Out Of Memory situation JVM is ended if not graceful 
> than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6777) Wrong reaction on Out Of Memory situation

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-6777:
-

 Summary: Wrong reaction on Out Of Memory situation
 Key: KAFKA-6777
 URL: https://issues.apache.org/jira/browse/KAFKA-6777
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 1.0.0
Reporter: Seweryn Habdank-Wojewodzki


Dears,

We already encountered many times problems related to Out Of Memory situation 
in Kafka Broker and streaming clients.

The scenario is the following.

When Kafka Broker (or Streaming Client) is under load and has too less memory, 
there are no errors in server logs. One can see some cryptic entries in GC 
logs, but they are definitely not self-explaining.

Kafka Broker (and Streaming Clients) works further. Later we see in JMX 
monitoring, that JVM uses more and more time in GC. In our case it grows from 
e. 1% to 80%-90% of CPU time is used by GC.

Next software collapses into zombie mode – process in not ending. In such a 
case I would expect, that process is crashing (e.g. got SIG SEGV). Even worse 
Kafka treats such a zombie process normal and somewhat sends messages, which 
are in fact getting lost, also the cluster is not excluding broken nodes. The 
question is how to configure Kafka to really terminate the JVM and not remain 
in zombie mode, to give a chance to other nodes to know, that something is dead.

I would expect that in Out Of Memory situation JVM is ended if not graceful 
than at least process is crashed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411602#comment-16411602
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6699:
---

[~asasvari] Topic:__consumer_offsetsPartitionCount:50   
ReplicationFactor:2 
Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer

> When one of two Kafka nodes are dead, streaming API cannot handle messaging
> ---
>
> Key: KAFKA-6699
> URL: https://issues.apache.org/jira/browse/KAFKA-6699
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> I am observing quite often, when Kafka Broker is partly dead(*), then 
> application, which uses streaming API are doing nothing.
> (*) Partly dead in my case it means that one of two Kafka nodes are out of 
> order. 
> Especially when disk is full on one machine, then Broker is going in some 
> strange state, where streaming API goes vacations. It seems like regular 
> producer/consumer API has no problem in such a case.
> Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411274#comment-16411274
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6699 at 3/23/18 11:54 AM:
-

[~astubbs] Replication factor is 2. Topic has as well 10 partitions.

[~asasvari] I will try to re-test this case with DEBUG logs on streamming API 
level, but it will take me some time.


was (Author: habdank):
[~astubbs]Replication factor is 2. Topic has as well 10 partitions.
[~asasvari]I will try to re-test this case with DEBUG logs on streamming API 
level, but it will take me some time.

> When one of two Kafka nodes are dead, streaming API cannot handle messaging
> ---
>
> Key: KAFKA-6699
> URL: https://issues.apache.org/jira/browse/KAFKA-6699
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> I am observing quite often, when Kafka Broker is partly dead(*), then 
> application, which uses streaming API are doing nothing.
> (*) Partly dead in my case it means that one of two Kafka nodes are out of 
> order. 
> Especially when disk is full on one machine, then Broker is going in some 
> strange state, where streaming API goes vacations. It seems like regular 
> producer/consumer API has no problem in such a case.
> Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411274#comment-16411274
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6699:
---

[~astubbs]Replication factor is 2. Topic has as well 10 partitions.
[~asasvari]I will try to re-test this case with DEBUG logs on streamming API 
level, but it will take me some time.

> When one of two Kafka nodes are dead, streaming API cannot handle messaging
> ---
>
> Key: KAFKA-6699
> URL: https://issues.apache.org/jira/browse/KAFKA-6699
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> Dears,
> I am observing quite often, when Kafka Broker is partly dead(*), then 
> application, which uses streaming API are doing nothing.
> (*) Partly dead in my case it means that one of two Kafka nodes are out of 
> order. 
> Especially when disk is full on one machine, then Broker is going in some 
> strange state, where streaming API goes vacations. It seems like regular 
> producer/consumer API has no problem in such a case.
> Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6699) When one of two Kafka nodes are dead, streaming API cannot handle messaging

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-6699:
-

 Summary: When one of two Kafka nodes are dead, streaming API 
cannot handle messaging
 Key: KAFKA-6699
 URL: https://issues.apache.org/jira/browse/KAFKA-6699
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.2
Reporter: Seweryn Habdank-Wojewodzki


Dears,

I am observing quite often, when Kafka Broker is partly dead(*), then 
application, which uses streaming API are doing nothing.

(*) Partly dead in my case it means that one of two Kafka nodes are out of 
order. 

Especially when disk is full on one machine, then Broker is going in some 
strange state, where streaming API goes vacations. It seems like regular 
producer/consumer API has no problem in such a case.

Can you have a look on that matter?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400055#comment-16400055
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-4392:
---

I see it on my Windows workstation as well.

The workaround is before kafka start to clear all files, which are stored in 
this folder like _/data/1/kafka-streams/myapp-streams/_ .

This is only workaround, because if the service is really working like 24/7 
then it will loos complete state and messages might be lost :-(.


> Failed to lock the state directory due to an unexpected exception
> -
>
> Key: KAFKA-4392
> URL: https://issues.apache.org/jira/browse/KAFKA-4392
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Ara Ebrahimi
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 0.10.2.0
>
>
> This happened on streaming startup, on a clean installation, no existing 
> folder. Here I was starting 4 instances of our streaming app on 4 machines 
> and one threw this exception. Seems to me there’s a race condition somewhere 
> when instances discover others, or something like that.
> 2016-11-02 15:43:47 INFO  StreamRunner:59 - Started http server successfully.
> 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state 
> directory due to an unexpected exception
> java.nio.file.NoSuchFileException: 
> /data/1/kafka-streams/myapp-streams/7_21/.lock
>   at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>   at 
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>   at java.nio.channels.FileChannel.open(FileChannel.java:287)
>   at java.nio.channels.FileChannel.open(FileChannel.java:335)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90)
>   at 
> org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
> ^C
> [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/
> ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or 
> directory
> [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/
> total 4
> drwxr-xr-x 74 root root 4096 Nov  2 15:44 .
> drwxr-xr-x  3 root root   27 Nov  2 15:43 ..
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_1
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_13
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_14
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_16
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_2
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_22
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_28
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_3
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_31
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_5
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_7
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_8
> drwxr-xr-x  3 root root   32 Nov  2 15:43 0_9
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_1
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_10
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_14
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_15
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_16
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_17
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_18
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_3
> drwxr-xr-x  3 root root   32 Nov  2 15:43 1_5
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_1
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_10
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_12
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_20
> drwxr-xr-x  3 root root   60 Nov  2 15:43 2_24
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_10
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_11
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_19
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_20
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_25
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_26
> drwxr-xr-x  3 root root   61 Nov  2 15:43 3_3
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_11
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_12
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_18
> drwxr-xr-x  3 root root   64 Nov  2 15:43 4_19
> drwxr-xr-x  3 root root   

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346563#comment-16346563
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 1/31/18 10:40 AM:
-

Answers:
 # v. 1.0.0 it is exatly what states in JIRA. The problem specification 
originates in 1.0.0.
 # I had downgraded the code, and was stable for along time, but yesterday I 
recognize, that it is also crashing, but not that easy as 1.0.0. I had descibed 
above how I reproduced in my environment the crashes. The main point is to run 
streamming application for a longer period of time, perhaps depending on 
retention time of the topics (my configuration is 2 days).


was (Author: habdank):
Answers:
 # v. 1.0.0 it is exatly what is in JIRA. The problem specification originates 
in 1.0.0.
 # I had down graded the code, and was stable for along time, but yesterday I 
recognize, that it is also crashing, but not that easy as 1.0.0. I had descibed 
above how I reproduced in my environment the crashes. The main point is to run 
streamming application longer peroiod of time, perhaps depending on retention 
time of the topics (my configuration is 2 days).

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346563#comment-16346563
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

Answers:
 # v. 1.0.0 it is exatly what is in JIRA. The problem specification originates 
in 1.0.0.
 # I had down graded the code, and was stable for along time, but yesterday I 
recognize, that it is also crashing, but not that easy as 1.0.0. I had descibed 
above how I reproduced in my environment the crashes. The main point is to run 
streamming application longer peroiod of time, perhaps depending on retention 
time of the topics (my configuration is 2 days).

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345235#comment-16345235
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 1/30/18 3:36 PM:


One more comment when it happens quite deterministic. When process _*A*_, which 
consumes messages from topic _input_ is already running many days and it is 
stopped while process _*B*_, which is consuming the same topic also is working 
many days, and then by starting again _*A*_ this error comes over and over 
again.
Current workaround is to stop both processes and start them again, that makes 
reballancing to be passed and no "null" exception is comming. It seems like 
some long term state is already deleted from topics, but it is hold in 
processes/borker and the new process, which is starting becomes some state with 
deleted items.


was (Author: habdank):
One more comment when it happens quite deterministic. When process _*A*_, which 
consumes messages from topic _input _is already running many days and it is 
stopped while process _*B*_, which is consuming the same topic also is working 
many days, and then by starting again _*A*_ this error comes over and over 
again.
Current workaround is to stop both processes and start them again, that makes 
reballancing to be passed and no "null" exception is comming. It seems like 
some long term state is already deleted from topics, but it is hold in 
processes/borker and the new process, which is starting becomes some state with 
deleted items.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing 

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345235#comment-16345235
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

One more comment when it happens quite deterministic. When process _*A*_, which 
consumes messages from topic _input _is already running many days and it is 
stopped while process _*B*_, which is consuming the same topic also is working 
many days, and then by starting again _*A*_ this error comes over and over 
again.
Current workaround is to stop both processes and start them again, that makes 
reballancing to be passed and no "null" exception is comming. It seems like 
some long term state is already deleted from topics, but it is hold in 
processes/borker and the new process, which is starting becomes some state with 
deleted items.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345209#comment-16345209
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

At 0.11.0.2 we got that as well:

{code}
2018-01-30 16:18:58 [my] [ERROR] StreamThread:192 - stream-thread 
[restreamer-my-19247358-945f-4415-8cac-92e51fef7690-StreamThread-1] Error 
caught during partition assignment, will abort the current process and re-throw 
at the end of rebalance: null
{code}

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345209#comment-16345209
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 1/30/18 3:21 PM:


At 0.11.0.2 we got that as well:

{code}
2018-01-30 16:18:58 [my] [ERROR] StreamThread:192 - stream-thread 
[streamer-my-19247358-945f-4415-8cac-92e51fef7690-StreamThread-1] Error caught 
during partition assignment, will abort the current process and re-throw at the 
end of rebalance: null
{code}


was (Author: habdank):
At 0.11.0.2 we got that as well:

{code}
2018-01-30 16:18:58 [my] [ERROR] StreamThread:192 - stream-thread 
[restreamer-my-19247358-945f-4415-8cac-92e51fef7690-StreamThread-1] Error 
caught during partition assignment, will abort the current process and re-throw 
at the end of rebalance: null
{code}

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki updated KAFKA-5882:
--
Affects Version/s: 0.11.0.1
   0.11.0.2

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6457) Error: NOT_LEADER_FOR_PARTITION leads to NPE

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337600#comment-16337600
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6457:
---

Questions:

AbstractCoordinator line 363:
RequestFuture future = initiateJoinGroup();
client.poll(future);

How it is guarantied, that future is never null?

line 402:
joinFuture = sendJoinGroupRequest();

How it is guarantied that joinFuture is not null?

> Error: NOT_LEADER_FOR_PARTITION leads to NPE
> 
>
> Key: KAFKA-6457
> URL: https://issues.apache.org/jira/browse/KAFKA-6457
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> One of our nodes was dead. Then the second one tooks all responsibility.
> But streamming aplication in the meanwhile crashed due to NPE caused by 
> {{Error: NOT_LEADER_FOR_PARTITION}}.
> The stack trace is below.
>  
> Is it something expected?
>  
> {code:java}
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer ...2018-01-17 
> 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-5, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-7, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [ERROR] AbstractCoordinator:296 - [Consumer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-consumer,
>  groupId=restreamer-my] Heartbeat thread for group restreamer-my failed due 
> to unexpected error
> java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436) 
> ~[my-restreamer.jar:?]
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:395) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934)
>  [my-restreamer.jar:?]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6457) Error: NOT_LEADER_FOR_PARTITION leads to NPE

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337422#comment-16337422
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6457:
---

All those tickets: KAFKA-6459, KAFKA-6457, KAFKA-5882  might be connected as 
more less something is wrong when rebalancing (or new leader is voted) happens 
in AbstractCoordinator. There are some places in code where objects are used, 
but they are {{null}}. And this class is common for all my stack traces. It is 
ofcourse question if AbstractCoordinator wrongly handles {{null}} or null shall 
never appead in those places, but the underlying class, which provides objects, 
deletes/resets to null them and shall not provided those objects.

> Error: NOT_LEADER_FOR_PARTITION leads to NPE
> 
>
> Key: KAFKA-6457
> URL: https://issues.apache.org/jira/browse/KAFKA-6457
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 1.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Major
>
> One of our nodes was dead. Then the second one tooks all responsibility.
> But streamming aplication in the meanwhile crashed due to NPE caused by 
> {{Error: NOT_LEADER_FOR_PARTITION}}.
> The stack trace is below.
>  
> Is it something expected?
>  
> {code:java}
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer ...2018-01-17 
> 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-5, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
>  Got error produce response with correlation id 768962 on topic-partition 
> my_internal_topic-7, retrying (9 attempts left). Error: 
> NOT_LEADER_FOR_PARTITION
> 2018-01-17 11:47:21 [my] [ERROR] AbstractCoordinator:296 - [Consumer 
> clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-consumer,
>  groupId=restreamer-my] Heartbeat thread for group restreamer-my failed due 
> to unexpected error
> java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436) 
> ~[my-restreamer.jar:?]
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:395) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) 
> ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275)
>  ~[my-restreamer.jar:?]
>     at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934)
>  [my-restreamer.jar:?]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6457) Error: NOT_LEADER_FOR_PARTITION leads to NPE

[Seweryn Habdank-Wojewodzki (JIRA)]

Seweryn Habdank-Wojewodzki created KAFKA-6457:
-

 Summary: Error: NOT_LEADER_FOR_PARTITION leads to NPE
 Key: KAFKA-6457
 URL: https://issues.apache.org/jira/browse/KAFKA-6457
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Seweryn Habdank-Wojewodzki


One of our nodes was dead. Then the second one took all responsibility.

But streamming aplication in the meanwhile crashed due to NPE caused by 
{{Error: NOT_LEADER_FOR_PARTITION}}.

The stack trace is below.

 

Is it something expected?

 

{code}

2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer ...2018-01-17 11:47:21 
[my] [WARN ] Sender:251 - [Producer 
clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
 Got error produce response with correlation id 768962 on topic-partition 
my_internal_topic-5, retrying (9 attempts left). Error: NOT_LEADER_FOR_PARTITION
2018-01-17 11:47:21 [my] [WARN ] Sender:251 - [Producer 
clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-producer]
 Got error produce response with correlation id 768962 on topic-partition 
my_internal_topic-7, retrying (9 attempts left). Error: NOT_LEADER_FOR_PARTITION
2018-01-17 11:47:21 [my] [ERROR] AbstractCoordinator:296 - [Consumer 
clientId=restreamer-my-fef07ca9-b067-45c0-a5af-68b5a1730dac-StreamThread-1-consumer,
 groupId=restreamer-my] Heartbeat thread for group restreamer-my failed due to 
unexpected error
java.lang.NullPointerException: null
    at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436) 
~[my-restreamer.jar:?]
    at org.apache.kafka.common.network.Selector.poll(Selector.java:395) 
~[my-restreamer.jar:?]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) 
~[my-restreamer.jar:?]
    at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238)
 ~[my-restreamer.jar:?]
    at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275)
 ~[my-restreamer.jar:?]
    at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934)
 [my-restreamer.jar:?]

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4315) Kafka Connect documentation problems

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki resolved KAFKA-4315.
---
Resolution: Done

I do not care anymore about this matter.

> Kafka Connect documentation problems
> 
>
> Key: KAFKA-4315
> URL: https://issues.apache.org/jira/browse/KAFKA-4315
> Project: Kafka
>  Issue Type: Bug
>Reporter: Seweryn Habdank-Wojewodzki
>
> On the base of documentation of the Kafka Connect - 
> http://kafka.apache.org/documentation#connect, I had tried to build example 
> in Java. It was not possible. 
> The code pieces available on the webpage are taken out of any context and 
> they are not compiling. 
> Also it seems they are taken completely from other code software parts, so 
> even putting them together shows, that they are not building any reasonable 
> example. And they tend to be very complex. where I would expect that the API 
> examples are driving "Hello World" like code.
> Also there are weak connections between examples from the Kafka documentation 
> and Kafka Connect tools code parts available in the Kafka source.
> Finally I would be nice to have a kind of statement in the Kafka 
> documentation which parts of API are stable and which are unstable or 
> experimental.
> I saw much (~20) of such a remarks in the Kafka code - I mean that API is 
> unstable. This note is very important, as we will plan additional effort to 
> prepare some facades for unstable code.
> In my opinion it is nothing wrong in experimental API, but all those matters 
> when documented shall be well documented. The current status of the main 
> Kafka documentation makes impression that Kafka Connect is well tested and 
> consistent and stable feature set, but it is not. What leads to confusion on 
> the effort management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (KAFKA-4315) Kafka Connect documentation problems

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-4315.
-

> Kafka Connect documentation problems
> 
>
> Key: KAFKA-4315
> URL: https://issues.apache.org/jira/browse/KAFKA-4315
> Project: Kafka
>  Issue Type: Bug
>Reporter: Seweryn Habdank-Wojewodzki
>
> On the base of documentation of the Kafka Connect - 
> http://kafka.apache.org/documentation#connect, I had tried to build example 
> in Java. It was not possible. 
> The code pieces available on the webpage are taken out of any context and 
> they are not compiling. 
> Also it seems they are taken completely from other code software parts, so 
> even putting them together shows, that they are not building any reasonable 
> example. And they tend to be very complex. where I would expect that the API 
> examples are driving "Hello World" like code.
> Also there are weak connections between examples from the Kafka documentation 
> and Kafka Connect tools code parts available in the Kafka source.
> Finally I would be nice to have a kind of statement in the Kafka 
> documentation which parts of API are stable and which are unstable or 
> experimental.
> I saw much (~20) of such a remarks in the Kafka code - I mean that API is 
> unstable. This note is very important, as we will plan additional effort to 
> prepare some facades for unstable code.
> In my opinion it is nothing wrong in experimental API, but all those matters 
> when documented shall be well documented. The current status of the main 
> Kafka documentation makes impression that Kafka Connect is well tested and 
> consistent and stable feature set, but it is not. What leads to confusion on 
> the effort management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (KAFKA-4908) consumer.properties logging warnings

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki closed KAFKA-4908.
-

> consumer.properties logging warnings
> 
>
> Key: KAFKA-4908
> URL: https://issues.apache.org/jira/browse/KAFKA-4908
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Minor
>
> default consumer.properties at startaup of the console consumer delivered 
> with Kafka package are logging warnings:
> [2017-03-15 16:36:57,439] WARN The configuration 
> 'zookeeper.connection.timeout.ms' was supplied but isn't a known config. 
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2017-03-15 16:36:57,455] WARN The configuration 'zookeeper.connect' was 
> supplied but isn't a known config. 
> (org.apache.kafka.clients.consumer.ConsumerConfig)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-4908) consumer.properties logging warnings

[Seweryn Habdank-Wojewodzki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Seweryn Habdank-Wojewodzki resolved KAFKA-4908.
---
Resolution: Done

Not an issue for me anymore.

> consumer.properties logging warnings
> 
>
> Key: KAFKA-4908
> URL: https://issues.apache.org/jira/browse/KAFKA-4908
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Minor
>
> default consumer.properties at startaup of the console consumer delivered 
> with Kafka package are logging warnings:
> [2017-03-15 16:36:57,439] WARN The configuration 
> 'zookeeper.connection.timeout.ms' was supplied but isn't a known config. 
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2017-03-15 16:36:57,455] WARN The configuration 'zookeeper.connect' was 
> supplied but isn't a known config. 
> (org.apache.kafka.clients.consumer.ConsumerConfig)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6260) AbstractCoordinator not clearly handles NULL Exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316179#comment-16316179
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6260:
---

Short question. When will those fixes released? :-)
Unfortunately at 
[Relase|https://issues.apache.org/jira/projects/KAFKA?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page]
 page there are no dates.

> AbstractCoordinator not clearly handles NULL Exception
> --
>
> Key: KAFKA-6260
> URL: https://issues.apache.org/jira/browse/KAFKA-6260
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RedHat Linux
>Reporter: Seweryn Habdank-Wojewodzki
>Assignee: Jason Gustafson
> Fix For: 1.1.0, 1.0.1
>
>
> The error reporting is not clear. But it seems that Kafka Heartbeat shuts 
> down application due to NULL exception caused by "fake" disconnections.
> One more comment. We are processing messages in the stream, but sometimes we 
> have to block processing for minutes, as consumers are not handling too much 
> load. Is it possibble that when stream is waiting, then heartbeat is as well 
> blocked?
> Can you check that?
> {code}
> 2017-11-23 23:54:47 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending Heartbeat request to coordinator 
> cljp01.eb.lan.at:9093 (id: 2147483646 rack: null)
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending HEARTBEAT 
> {group_id=kafka-endpoint,generation_id=3834,member_id=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer-94f18be5-e49a-4817-9e5a-fe82a64e0b08}
>  with correlation id 24 to node 2147483646
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Completed receive from node 2147483646 for HEARTBEAT 
> with correlation id 24, received {throttle_time_ms=0,error_code=0}
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:52 DEBUG NetworkClient:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Disconnecting from node 1 due to request timeout.
> 2017-11-23 23:54:52 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled request 
> {replica_id=-1,max_wait_time=6000,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=clj_internal_topic,partitions=[{partition=6,fetch_offset=211558395,log_start_offset=-1,max_bytes=1048576},{partition=8,fetch_offset=210178209,log_start_offset=-1,max_bytes=1048576},{partition=0,fetch_offset=209353523,log_start_offset=-1,max_bytes=1048576},{partition=2,fetch_offset=209291462,log_start_offset=-1,max_bytes=1048576},{partition=4,fetch_offset=210728595,log_start_offset=-1,max_bytes=1048576}]}]}
>  with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG ConsumerNetworkClient:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled FETCH request RequestHeader(apiKey=FETCH, 
> apiVersion=6, 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  correlationId=21) with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG Fetcher:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Fetch request 
> {clj_internal_topic-6=(offset=211558395, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-8=(offset=210178209, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-0=(offset=209353523, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-2=(offset=209291462, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-4=(offset=210728595, logStartOffset=-1, 
> maxBytes=1048576)} to cljp01.eb.lan.at:9093 (id: 1 rack: DC-1) failed 
> org.apache.kafka.common.errors.DisconnectException: null
> 2017-11-23 23:54:52 TRACE NetworkClient:123 - [Consumer 
> 

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281484#comment-16281484
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

Hmm... I will try to retest all together with fix for KAFKA-6260.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272747#comment-16272747
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 12/7/17 8:09 AM:


[~mjsax] In meanwhile I had ported the code to {{1.0.0}} :-). I will try do my 
best.


was (Author: habdank):
[~mjsax] In mean while I had ported the code to {{1.0.0}} :-). I will try do my 
best.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6260) AbstractCoordinator not clearly handles NULL Exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281481#comment-16281481
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6260:
---

Thanks a lot!

When releases will be ready, I will test them.
I am not sure if and how I can get lib earlier before they are relased :-).

> AbstractCoordinator not clearly handles NULL Exception
> --
>
> Key: KAFKA-6260
> URL: https://issues.apache.org/jira/browse/KAFKA-6260
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RedHat Linux
>Reporter: Seweryn Habdank-Wojewodzki
>Assignee: Jason Gustafson
> Fix For: 1.1.0, 1.0.1
>
>
> The error reporting is not clear. But it seems that Kafka Heartbeat shuts 
> down application due to NULL exception caused by "fake" disconnections.
> One more comment. We are processing messages in the stream, but sometimes we 
> have to block processing for minutes, as consumers are not handling too much 
> load. Is it possibble that when stream is waiting, then heartbeat is as well 
> blocked?
> Can you check that?
> {code}
> 2017-11-23 23:54:47 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending Heartbeat request to coordinator 
> cljp01.eb.lan.at:9093 (id: 2147483646 rack: null)
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending HEARTBEAT 
> {group_id=kafka-endpoint,generation_id=3834,member_id=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer-94f18be5-e49a-4817-9e5a-fe82a64e0b08}
>  with correlation id 24 to node 2147483646
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Completed receive from node 2147483646 for HEARTBEAT 
> with correlation id 24, received {throttle_time_ms=0,error_code=0}
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:52 DEBUG NetworkClient:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Disconnecting from node 1 due to request timeout.
> 2017-11-23 23:54:52 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled request 
> {replica_id=-1,max_wait_time=6000,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=clj_internal_topic,partitions=[{partition=6,fetch_offset=211558395,log_start_offset=-1,max_bytes=1048576},{partition=8,fetch_offset=210178209,log_start_offset=-1,max_bytes=1048576},{partition=0,fetch_offset=209353523,log_start_offset=-1,max_bytes=1048576},{partition=2,fetch_offset=209291462,log_start_offset=-1,max_bytes=1048576},{partition=4,fetch_offset=210728595,log_start_offset=-1,max_bytes=1048576}]}]}
>  with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG ConsumerNetworkClient:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled FETCH request RequestHeader(apiKey=FETCH, 
> apiVersion=6, 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  correlationId=21) with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG Fetcher:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Fetch request 
> {clj_internal_topic-6=(offset=211558395, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-8=(offset=210178209, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-0=(offset=209353523, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-2=(offset=209291462, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-4=(offset=210728595, logStartOffset=-1, 
> maxBytes=1048576)} to cljp01.eb.lan.at:9093 (id: 1 rack: DC-1) failed 
> org.apache.kafka.common.errors.DisconnectException: null
> 2017-11-23 23:54:52 TRACE NetworkClient:123 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272747#comment-16272747
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 11/30/17 2:47 PM:
-

[~mjsax] In mean while I had ported the code to {{1.0.0}} :-). I will try do my 
best.


was (Author: habdank):
[~mjsax] In mean while I had ported the code to {1.0.0} :-). I will try do my 
best.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272747#comment-16272747
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

[~mjsax] In mean while I had ported the code to {1.0.0} :-). I will try do my 
best.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-6260) AbstractCoordinator not clearly handles NULL Exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272741#comment-16272741
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6260 at 11/30/17 2:44 PM:
-

[~hachikuji] 
* Can you tell me exactly (exact setting name), which settings are logically 
combined, so I can set them respectively to our timings including our wait for 
end-clients results, please?
* Yes I can test any hot fixes, the only matter is how can I get them into our 
build process :-).



was (Author: habdank):
[~hachikuji] 
* Can you tell me exactly which settings are logically combined, so I can set 
them respectively to our timings including our wait for end-clients results, 
please?
* Yes I can test any hot fixes, the only matter is how can I get them into our 
build process :-).


> AbstractCoordinator not clearly handles NULL Exception
> --
>
> Key: KAFKA-6260
> URL: https://issues.apache.org/jira/browse/KAFKA-6260
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RedHat Linux
>Reporter: Seweryn Habdank-Wojewodzki
>Assignee: Jason Gustafson
> Fix For: 1.1.0, 1.0.1
>
>
> The error reporting is not clear. But it seems that Kafka Heartbeat shuts 
> down application due to NULL exception caused by "fake" disconnections.
> One more comment. We are processing messages in the stream, but sometimes we 
> have to block processing for minutes, as consumers are not handling too much 
> load. Is it possibble that when stream is waiting, then heartbeat is as well 
> blocked?
> Can you check that?
> {code}
> 2017-11-23 23:54:47 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending Heartbeat request to coordinator 
> cljp01.eb.lan.at:9093 (id: 2147483646 rack: null)
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending HEARTBEAT 
> {group_id=kafka-endpoint,generation_id=3834,member_id=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer-94f18be5-e49a-4817-9e5a-fe82a64e0b08}
>  with correlation id 24 to node 2147483646
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Completed receive from node 2147483646 for HEARTBEAT 
> with correlation id 24, received {throttle_time_ms=0,error_code=0}
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:52 DEBUG NetworkClient:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Disconnecting from node 1 due to request timeout.
> 2017-11-23 23:54:52 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled request 
> {replica_id=-1,max_wait_time=6000,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=clj_internal_topic,partitions=[{partition=6,fetch_offset=211558395,log_start_offset=-1,max_bytes=1048576},{partition=8,fetch_offset=210178209,log_start_offset=-1,max_bytes=1048576},{partition=0,fetch_offset=209353523,log_start_offset=-1,max_bytes=1048576},{partition=2,fetch_offset=209291462,log_start_offset=-1,max_bytes=1048576},{partition=4,fetch_offset=210728595,log_start_offset=-1,max_bytes=1048576}]}]}
>  with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG ConsumerNetworkClient:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled FETCH request RequestHeader(apiKey=FETCH, 
> apiVersion=6, 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  correlationId=21) with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG Fetcher:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Fetch request 
> {clj_internal_topic-6=(offset=211558395, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-8=(offset=210178209, logStartOffset=-1, 
> maxBytes=1048576), 

[jira] [Commented] (KAFKA-6260) AbstractCoordinator not clearly handles NULL Exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272741#comment-16272741
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6260:
---

[~hachikuji] 
* Can you tell me exactly which settings are logically combined, so I can set 
them respectively to our timings including our wait for end-clients results, 
please?
* Yes I can test any hot fixes, the only matter is how can I get them into our 
build process :-).


> AbstractCoordinator not clearly handles NULL Exception
> --
>
> Key: KAFKA-6260
> URL: https://issues.apache.org/jira/browse/KAFKA-6260
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RedHat Linux
>Reporter: Seweryn Habdank-Wojewodzki
>Assignee: Jason Gustafson
> Fix For: 1.1.0, 1.0.1
>
>
> The error reporting is not clear. But it seems that Kafka Heartbeat shuts 
> down application due to NULL exception caused by "fake" disconnections.
> One more comment. We are processing messages in the stream, but sometimes we 
> have to block processing for minutes, as consumers are not handling too much 
> load. Is it possibble that when stream is waiting, then heartbeat is as well 
> blocked?
> Can you check that?
> {code}
> 2017-11-23 23:54:47 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending Heartbeat request to coordinator 
> cljp01.eb.lan.at:9093 (id: 2147483646 rack: null)
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending HEARTBEAT 
> {group_id=kafka-endpoint,generation_id=3834,member_id=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer-94f18be5-e49a-4817-9e5a-fe82a64e0b08}
>  with correlation id 24 to node 2147483646
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Completed receive from node 2147483646 for HEARTBEAT 
> with correlation id 24, received {throttle_time_ms=0,error_code=0}
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:52 DEBUG NetworkClient:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Disconnecting from node 1 due to request timeout.
> 2017-11-23 23:54:52 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled request 
> {replica_id=-1,max_wait_time=6000,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=clj_internal_topic,partitions=[{partition=6,fetch_offset=211558395,log_start_offset=-1,max_bytes=1048576},{partition=8,fetch_offset=210178209,log_start_offset=-1,max_bytes=1048576},{partition=0,fetch_offset=209353523,log_start_offset=-1,max_bytes=1048576},{partition=2,fetch_offset=209291462,log_start_offset=-1,max_bytes=1048576},{partition=4,fetch_offset=210728595,log_start_offset=-1,max_bytes=1048576}]}]}
>  with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG ConsumerNetworkClient:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled FETCH request RequestHeader(apiKey=FETCH, 
> apiVersion=6, 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  correlationId=21) with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG Fetcher:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Fetch request 
> {clj_internal_topic-6=(offset=211558395, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-8=(offset=210178209, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-0=(offset=209353523, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-2=(offset=209291462, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-4=(offset=210728595, logStartOffset=-1, 
> maxBytes=1048576)} to cljp01.eb.lan.at:9093 (id: 1 rack: DC-1) failed 
> org.apache.kafka.common.errors.DisconnectException: null
> 2017-11-23 23:54:52 TRACE 

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270420#comment-16270420
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 11/29/17 8:58 AM:
-

[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}
By calling {{kBuilder.build()}} I have access to Topology, but not to 
TopologyBuilder (which is deprecated). 
Will I got, what you need by calling {{topology.toString()}}?

My code now looks like:
{code}
Topology kTopology = kBuilder.build();
LOG.info( "Topology: {}", kTopology.toString() );
streams = new KafkaStreams( kTopology, streamsConfig );
{code}
Shall I call that just before {{streams.start()}} or after?



was (Author: habdank):
[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}
By calling {{kBuilder.build()}} I have access to Topology, but not to 
TopologyBuilder (which is deprecated). 
Will I got, what you need by calling {{topology.toString()}}?

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by 

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270420#comment-16270420
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 11/29/17 8:55 AM:
-

[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}
By calling {{kBuilder.build()}} I have access to Topology, but not to 
TopologyBuilder (which is deprecated). 
Will I got, what you need by calling {{topology.toString()}}?


was (Author: habdank):
[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}
By calling {{kBuilder.build()}} I have access to Topology, but not to 
TopologyBuilder (which is deprecated). Will I got, what you need by calling 
{{topology.toString()}}?

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270420#comment-16270420
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 11/29/17 8:54 AM:
-

[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}
By calling {{kBuilder.build()}} I have access to Topology, but not to 
TopologyBuilder (which is deprecated). Will I got, what you need by calling 
{{topology.toString()}}?


was (Author: habdank):
[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}
By calling {{kBuilder.build()}} I have access to Topology, but not to 
TopologyBuilder.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270420#comment-16270420
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 11/29/17 8:52 AM:
-

[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}
By calling {{kBuilder.build()}} I have access to Topology, but not to 
TopologyBuilder.


was (Author: habdank):
[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270420#comment-16270420
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: `KStream stringInput = 
kBuilder.stream( inTopicName );` and `streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );`

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270420#comment-16270420
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-5882 at 11/29/17 8:49 AM:
-

[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: {{KStream stringInput = 
kBuilder.stream( inTopicName );}} and {{streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );}}


was (Author: habdank):
[~mjsax] How can I get result similar to 
"topologyBuilder.build(null).toString()", if I am not using topology builder? 
I am creating streams using: `KStream stringInput = 
kBuilder.stream( inTopicName );` and `streams = new KafkaStreams( 
kBuilder.build(), streamsConfig );`

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6260) AbstractCoordinator not clearly handles NULL Exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270342#comment-16270342
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6260:
---

BTW the other bug: KAFKA-5882 happens also in the same environment.

> AbstractCoordinator not clearly handles NULL Exception
> --
>
> Key: KAFKA-6260
> URL: https://issues.apache.org/jira/browse/KAFKA-6260
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RedHat Linux
>Reporter: Seweryn Habdank-Wojewodzki
>
> The error reporting is not clear. But it seems that Kafka Heartbeat shuts 
> down application due to NULL exception caused by "fake" disconnections.
> One more comment. We are processing messages in the stream, but sometimes we 
> have to block processing for minutes, as consumers are not handling too much 
> load. Is it possibble that when stream is waiting, then heartbeat is as well 
> blocked?
> Can you check that?
> {code}
> 2017-11-23 23:54:47 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending Heartbeat request to coordinator 
> cljp01.eb.lan.at:9093 (id: 2147483646 rack: null)
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending HEARTBEAT 
> {group_id=kafka-endpoint,generation_id=3834,member_id=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer-94f18be5-e49a-4817-9e5a-fe82a64e0b08}
>  with correlation id 24 to node 2147483646
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Completed receive from node 2147483646 for HEARTBEAT 
> with correlation id 24, received {throttle_time_ms=0,error_code=0}
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:52 DEBUG NetworkClient:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Disconnecting from node 1 due to request timeout.
> 2017-11-23 23:54:52 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled request 
> {replica_id=-1,max_wait_time=6000,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=clj_internal_topic,partitions=[{partition=6,fetch_offset=211558395,log_start_offset=-1,max_bytes=1048576},{partition=8,fetch_offset=210178209,log_start_offset=-1,max_bytes=1048576},{partition=0,fetch_offset=209353523,log_start_offset=-1,max_bytes=1048576},{partition=2,fetch_offset=209291462,log_start_offset=-1,max_bytes=1048576},{partition=4,fetch_offset=210728595,log_start_offset=-1,max_bytes=1048576}]}]}
>  with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG ConsumerNetworkClient:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled FETCH request RequestHeader(apiKey=FETCH, 
> apiVersion=6, 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  correlationId=21) with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG Fetcher:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Fetch request 
> {clj_internal_topic-6=(offset=211558395, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-8=(offset=210178209, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-0=(offset=209353523, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-2=(offset=209291462, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-4=(offset=210728595, logStartOffset=-1, 
> maxBytes=1048576)} to cljp01.eb.lan.at:9093 (id: 1 rack: DC-1) failed 
> org.apache.kafka.common.errors.DisconnectException: null
> 2017-11-23 23:54:52 TRACE NetworkClient:123 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Found least loaded node cljp01.eb.lan.at:9093 (id: 1 
> rack: DC-1)
> 2017-11-23 23:54:52 DEBUG NetworkClient:183 - [Consumer 
> 

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270341#comment-16270341
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

BTW the other bug: KAFKA-6260 happens also in the same environment.

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5882) NullPointerException in StreamTask

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270338#comment-16270338
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-5882:
---

[~mjsax] What do you mean by "DEBUG logs of Streams"? Which namespaces or 
classes shall I log in debug. Do you need:
{code}

  

{code}
Or something other?

> NullPointerException in StreamTask
> --
>
> Key: KAFKA-5882
> URL: https://issues.apache.org/jira/browse/KAFKA-5882
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Seweryn Habdank-Wojewodzki
>
> It seems bugfix [KAFKA-5073|https://issues.apache.org/jira/browse/KAFKA-5073] 
> is made, but introduce some other issue.
> In some cases (I am not sure which ones) I got NPE (below).
> I would expect that even in case of FATAL error anythink except NPE is thrown.
> {code}
> 2017-09-12 23:34:54 ERROR ConsumerCoordinator:269 - User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener 
> for group streamer failed on partition assignment
> java.lang.NullPointerException: null
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:123)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:1234)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:294)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:254)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:1313)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$1100(StreamThread.java:73)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsAssigned(StreamThread.java:183)
>  ~[myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:265)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:363)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:297)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1078)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043) 
> [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:582)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
>  [myapp-streamer.jar:?]
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527)
>  [myapp-streamer.jar:?]
> 2017-09-12 23:34:54 INFO  StreamThread:1040 - stream-thread 
> [streamer-3a44578b-faa8-4b5b-bbeb-7a7f04639563-StreamThread-1] Shutting down
> 2017-09-12 23:34:54 INFO  KafkaProducer:972 - Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-6260) AbstractCoordinator not clearly handles NULL Exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266883#comment-16266883
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-6260:
---

[~ijuma] Include some reproducible example is pretty hard for me, as we have 
some servers and at two of them this happend (continuously). 
The other are working well, but the definitive difference is load. At those two 
we have pretty high load in comparison to others.
Also the problem is that our Kafka uses SSL and ACLs, which are changing timing 
and this part I cannot attach.
Kafka and Log4j settings I had already included.

Kafka code it self is pretty easy. It looks like (a bit compacted, as we have 
it spread in some methods):

{code}
StreamsBuilder kBuilder = new StreamsBuilder();
this.kafkaAndStreamsProperties = kafkaAndStreamsProperties;

this.kafkaAndStreamsProperties.put( 
StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass() );
this.kafkaAndStreamsProperties.put( 
StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass() );

StreamsConfig streamsConfig = new StreamsConfig( kafkaAndStreamsProperties );
KafkaInputConfiguration kic = new KafkaInputConfiguration();

final String inTopicName = kic.getInputTopicParameters().getTopicName();

KStream stringInput = kBuilder.stream( inTopicName );

stringInput.foreach( ( k, v ) -> {
try {
MyRecordRto lrr = objectMapper.readValue( v, MyRecordRto.class 
);

// it could take much time as sometimes we are wating here 
until the processors are finishing own work
// at high load here we could wait up to minute
myStreamProcessor.process( lrr );

} catch ( IOException pe ) {
LOG.warn( "Parsing JSON encountered error: {}", pe.getMessage() 
);
LOG.trace( "Error: {}", pe );
} catch ( Exception e ) {
LOG.warn( "Processing message encountered error: {}", 
e.getMessage() );
LOG.trace( "Error: {}", e );
}
} );

KafkaStreams streams = new KafkaStreams( kBuilder.build(), streamsConfig );
streams.setUncaughtExceptionHandler( ( Thread t, Throwable e ) -> {
synchronized ( this ) {
LOG.fatal( "Caught unhandled exception: {}; {} 
in thread {}",
e.getMessage(), 
e.getStackTrace(), t.getName() );
this.stop( 5L );
// seems to be asymmetric shutdown hook shall 
not contains System.exit()
// for v. 0.11.0.0 see official report 
KAFKA-4366
// but setUncaughtExceptionHandler without 
System.exit() hangs.
System.exit( 1 );
}
}
);
streams.cleanUp();
streams.start();
{code}

I have new trace level logs. I will rework them and attach.


> AbstractCoordinator not clearly handles NULL Exception
> --
>
> Key: KAFKA-6260
> URL: https://issues.apache.org/jira/browse/KAFKA-6260
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RedHat Linux
>Reporter: Seweryn Habdank-Wojewodzki
>
> The error reporting is not clear. But it seems that Kafka Heartbeat shuts 
> down application due to NULL exception caused by "fake" disconnections.
> One more comment. We are processing messages in the stream, but sometimes we 
> have to block processing for minutes, as consumers are not handling too much 
> load. Is it possibble that when stream is waiting, then heartbeat is as well 
> blocked?
> Can you check that?
> {code}
> 2017-11-23 23:54:47 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending Heartbeat request to coordinator 
> cljp01.eb.lan.at:9093 (id: 2147483646 rack: null)
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending HEARTBEAT 
> {group_id=kafka-endpoint,generation_id=3834,member_id=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer-94f18be5-e49a-4817-9e5a-fe82a64e0b08}
>  with correlation id 24 to node 2147483646
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  

[jira] [Comment Edited] (KAFKA-6260) AbstractCoordinator not clearly handles NULL Exception

[Seweryn Habdank-Wojewodzki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265125#comment-16265125
 ] 

Seweryn Habdank-Wojewodzki edited comment on KAFKA-6260 at 11/24/17 2:13 PM:
-

Thanks for the response. I had updated content of the ticket. We are using 
Kafka 1.0.0 with SSL - settings dumped from logs are in ticket.


was (Author: habdank):
Thanks for the responce. I had updated content of the ticket. We are using 
Kafka 1.0.0 with SSL - settings dumped from logs are in ticket.

> AbstractCoordinator not clearly handles NULL Exception
> --
>
> Key: KAFKA-6260
> URL: https://issues.apache.org/jira/browse/KAFKA-6260
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RedHat Linux
>Reporter: Seweryn Habdank-Wojewodzki
>
> The error reporting is not clear. But it seems that Kafka Heartbeat shuts 
> down application due to NULL exception caused by "fake" disconnections.
> One more comment. We are processing messages in the stream, but sometimes we 
> have to block processing for minutes, as consumers are not handling too much 
> load. Is it possibble that when stream is waiting, then heartbeat is as well 
> blocked?
> Can you check that?
> {code}
> 2017-11-23 23:54:47 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending Heartbeat request to coordinator 
> cljp01.eb.lan.at:9093 (id: 2147483646 rack: null)
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Sending HEARTBEAT 
> {group_id=kafka-endpoint,generation_id=3834,member_id=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer-94f18be5-e49a-4817-9e5a-fe82a64e0b08}
>  with correlation id 24 to node 2147483646
> 2017-11-23 23:54:50 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Completed receive from node 2147483646 for HEARTBEAT 
> with correlation id 24, received {throttle_time_ms=0,error_code=0}
> 2017-11-23 23:54:50 DEBUG AbstractCoordinator:177 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Received successful Heartbeat response
> 2017-11-23 23:54:52 DEBUG NetworkClient:183 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Disconnecting from node 1 due to request timeout.
> 2017-11-23 23:54:52 TRACE NetworkClient:135 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled request 
> {replica_id=-1,max_wait_time=6000,min_bytes=1,max_bytes=52428800,isolation_level=0,topics=[{topic=clj_internal_topic,partitions=[{partition=6,fetch_offset=211558395,log_start_offset=-1,max_bytes=1048576},{partition=8,fetch_offset=210178209,log_start_offset=-1,max_bytes=1048576},{partition=0,fetch_offset=209353523,log_start_offset=-1,max_bytes=1048576},{partition=2,fetch_offset=209291462,log_start_offset=-1,max_bytes=1048576},{partition=4,fetch_offset=210728595,log_start_offset=-1,max_bytes=1048576}]}]}
>  with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG ConsumerNetworkClient:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Cancelled FETCH request RequestHeader(apiKey=FETCH, 
> apiVersion=6, 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  correlationId=21) with correlation id 21 due to node 1 being disconnected
> 2017-11-23 23:54:52 DEBUG Fetcher:195 - [Consumer 
> clientId=kafka-endpoint-be51569b-8795-4709-8ec8-28c9cd099a31-StreamThread-1-consumer,
>  groupId=kafka-endpoint] Fetch request 
> {clj_internal_topic-6=(offset=211558395, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-8=(offset=210178209, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-0=(offset=209353523, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-2=(offset=209291462, logStartOffset=-1, 
> maxBytes=1048576), clj_internal_topic-4=(offset=210728595, logStartOffset=-1, 
> maxBytes=1048576)} to cljp01.eb.lan.at:9093 (id: 1 rack: DC-1) failed 
> org.apache.kafka.common.errors.DisconnectException: null
> 2017-11-23 23:54:52 TRACE