pdeva commented on issue #6166: MAJOR issue: Unrecoverable error in KIS 0.12.2 and 0.12.1 URL: https://github.com/apache/incubator-druid/issues/6166#issuecomment-412628004 so i just tried increasing the size of the faulty MM instance to an absolutely absurd amount of cpus, and that seems to have done the trick. Here is what i think is happening: 1. When the node is started, it tries to start all the KI tasks. 2. Since the tasks have been down for a while, they saturate all cpu trying to read and process data from kafka. 3. due to cpu saturation, they cannot answer to overlord's status checks. 4. since overlord's status check fails, it kills the tasks and respawns them, thus creating an endless loop. in fact, even with the mighty powerful cpu box the status check from overlord failed a few times (but it seems to have passed before timing out completely). the solution would be to: 1. increase overlord timeout for individual status pings to tasks (not sure if there is an existing setting for this) 2. increase the total time and number of retries before giving up. this issue seems to be related to #6117 and #5340 for occuring in the first place. connection problems across nodes are a huge issue with druid right now.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
