[
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605851#comment-16605851
]
Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/6/18 2:42 PM:
---------------------------------------------------------------------------
Problem is that KSTREAM-SOURCE-XXXXXXXXX is mostly KSTREAM-SOURCE-000000000
independently of which process and how much processes are running (or trying to
run).
How I reproduce error at my side. Let's assume I have low message flow < 100
Msg/sec Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.
At second server I am starting. second instance. The same. It is starting.
The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec
Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even if memory spec
stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB
leads at some point to yet another mystic crasches.
At second server I am starting second instance. If I am starting it with 300 MB
- I got immediately this error. Application tries to start, but then I got this
error and all affected topics are going to be dead. If I am giving 1GB, it is
better application works some hours, but any minimal peak aroud 5000 Msg/s to
e.g. 7000 Msg/s, causes the same.
Finally - now - I am starting processes with 5GB. Processes could work
continuously like 2-4 days.
I am sorry I have no better description of the problem and visible error.
Once I tried to start TRACE level logs in Kafka, but this is impossible with
message flow at level of 5000 Msg/s.
was (Author: habdank):
Problem is that KSTREAM-SOURCE-XXXXXXXXX is mostly KSTREAM-SOURCE-000000000
independently of which process and how much processes are running (or trying to
run).
How I reproduce error at my side. Let's assume I have low message flow < 100
Msg/sec Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.
At second server I am starting. second instance. The same. It is starting.
The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec
Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even if memory spec
stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB
leads at some point to yet another mystic crasches.
At second server I am starting. second instance. If I am starting it with 300
MB - I got immediately this error. Application tries to starrt, but then I got
this error and all affected topics are goig to be dead. If I am giving 1GB, it
is better application works some hours, but any minimal peak aroud 5000 Msg/s
to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes
with 5GB. they could work continuously like 2-4 days.
I am sorry I have no better description.
Once I tried to start TRACE level logs in Kafka, but this is impossible with
message flow at 5000 Msg/s.
> Mystic FATAL error
> ------------------
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.11.0.3, 1.1.1
> Reporter: Seweryn Habdank-Wojewodzki
> Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-0000000000,
> topic=my_instance_medium_topic, partition=1, offset=198900203;
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
> in thread
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
> State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
