[
https://issues.apache.org/jira/browse/CASSANDRA-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791547#comment-14791547
]
Stefania commented on CASSANDRA-10321:
--------------------------------------
Brandon is correct, we need a thread dump or some profiling information to find
out what threads were using the cpu. The message in the logs is simply saying,
there are still 32 pending gossip tasks, I am going to sleep for 100
milliseconds and then check again. And as you can see it makes no progress so
something else is blocking the message processing threads. Are there any
specific events that trigger this?
If you can reproduce it, you can attach [visual vm|https://visualvm.java.net/]
(cpu sampler is best for this type of cases) or [java flight
recorder|http://www.oracle.com/technetwork/java/javaseproducts/mission-control/java-mission-control-1998576.html]
or launch _jstack pid_ several times.
> Gossip to dead nodes caused CPU usage to be 100%
> -------------------------------------------------
>
> Key: CASSANDRA-10321
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10321
> Project: Cassandra
> Issue Type: Bug
> Reporter: Dikang Gu
>
> For one node, the cpu usage jumped to 100%, and logs are full of:
> 2015-09-14_16:34:45.56407 WARN 16:34:45 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:46.66616 WARN 16:34:46 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:47.76830 WARN 16:34:47 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:48.87043 WARN 16:34:48 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:49.97253 WARN 16:34:49 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:51.07462 WARN 16:34:51 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:52.17669 WARN 16:34:52 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:53.27880 WARN 16:34:53 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:54.38090 WARN 16:34:54 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:55.48301 WARN 16:34:55 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:56.58509 WARN 16:34:56 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:57.68721 WARN 16:34:57 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:58.78932 WARN 16:34:58 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:59.89142 WARN 16:34:59 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:00.99352 WARN 16:35:00 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:02.09563 WARN 16:35:02 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:03.19775 WARN 16:35:03 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:04.29982 WARN 16:35:04 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:05.40187 WARN 16:35:05 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:06.50369 WARN 16:35:06 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:07.60577 WARN 16:35:07 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:08.70779 WARN 16:35:08 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:09.80968 WARN 16:35:09 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:10.91157 WARN 16:35:10 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:12.01365 WARN 16:35:12 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:13.11569 WARN 16:35:13 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:14.21757 WARN 16:35:14 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:15.31942 WARN 16:35:15 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:16.42132 WARN 16:35:16 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:17.52332 WARN 16:35:17 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:18.62511 WARN 16:35:18 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:19.72697 WARN 16:35:19 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:20.82872 WARN 16:35:20 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:21.93074 WARN 16:35:21 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:23.03281 WARN 16:35:23 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:24.13478 WARN 16:35:24 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:25.23672 WARN 16:35:25 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:26.33862 WARN 16:35:26 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:27.44068 WARN 16:35:27 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:28.54267 WARN 16:35:28 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:29.64461 WARN 16:35:29 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:30.74646 WARN 16:35:30 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:31.84823 WARN 16:35:31 Gossip stage has 32 pending tasks;
> skipping status check (no nodes will be marked down)
> 2015-09-14_16:40:09.97371 WARN 16:40:09 /2401:db00:2020:716b:face:0:21:0 has
> 2311 dropped hints, because node is down past configured hint window.
> This happened to several nodes in our cluster, but one node each time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
