[ https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606846#comment-16606846 ]
Seweryn Habdank-Wojewodzki commented on KAFKA-7214: --------------------------------------------------- Hi [~vvcephei], This is what I had stayed in other problem reports. Memory consumption or memory model described in Kafka documentation does not fit to reality. Now I am in the phase, when I am fully guessing, by obtaining mystic fatals during data processing. Is it expected, to increase memory when I get any kind of error in Kafka? I would be really greatful when I can more less, even with 30% overhead, but calculate, how much memory I need for my service to process X Msg/s with given size and given retention and given whatever. But it is not the case. And again. If error reporting would be -> Out Of Memory. I would also quickly see that is is really memory issue and I would be able to calculate this myself. But ending with: {code} 2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close task due to the following error: org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0 {code} Does not even point to the problem, and still I am not sure if it is really memory problem, I see only, that when I give more memory, I do not see it that often. > Mystic FATAL error > ------------------ > > Key: KAFKA-7214 > URL: https://issues.apache.org/jira/browse/KAFKA-7214 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 0.11.0.3, 1.1.1 > Reporter: Seweryn Habdank-Wojewodzki > Priority: Critical > > Dears, > Very often at startup of the streaming application I got exception: > {code} > Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-0000000000, > topic=my_instance_medium_topic, partition=1, offset=198900203; > [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212), > > org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347), > > org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420), > > org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339), > > org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648), > > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513), > > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482), > > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)] > in thread > my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62 > {code} > and then (without shutdown request from my side): > {code} > 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread > [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62] > State transition from PENDING_SHUTDOWN to DEAD. > {code} > What is this? > How to correctly handle it? > Thanks in advance for help. -- This message was sent by Atlassian JIRA (v7.6.3#76005)