[
https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194091#comment-15194091
]
Ruoran Wang commented on CASSANDRA-9625:
----------------------------------------
Here are the thread-dump result. First one is when the reporter is still
working, the second one is when reporter is stopped.
{noformat}
"metrics-graphite-reporter-thread-1" #574 daemon prio=5 os_prio=0
tid=0x00007fae39b21800 nid=0x4940 waiting on condition [0x00007fa571910000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fa67d7972d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
{noformat}
"metrics-graphite-reporter-thread-1" #555 daemon prio=5 os_prio=0
tid=0x00007fdf4e7f7800 nid=0xe43 waiting for monitor entry [0x00007fd6bb86b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getEstimatedRemainingTasks(WrappingCompactionStrategy.java:162)
- waiting to lock <0x00007fd72ced3e38> (a
org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
at
org.apache.cassandra.metrics.ColumnFamilyMetrics$13.value(ColumnFamilyMetrics.java:357)
at
org.apache.cassandra.metrics.ColumnFamilyMetrics$13.value(ColumnFamilyMetrics.java:354)
at
org.apache.cassandra.metrics.ColumnFamilyMetrics$33.value(ColumnFamilyMetrics.java:662)
at
org.apache.cassandra.metrics.ColumnFamilyMetrics$33.value(ColumnFamilyMetrics.java:656)
at
com.yammer.metrics.reporting.GraphiteReporter.processGauge(GraphiteReporter.java:304)
at
com.yammer.metrics.reporting.GraphiteReporter.processGauge(GraphiteReporter.java:26)
at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28)
at
com.yammer.metrics.reporting.GraphiteReporter.printRegularMetrics(GraphiteReporter.java:247)
at
com.yammer.metrics.reporting.GraphiteReporter.run(GraphiteReporter.java:213)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> GraphiteReporter not reporting
> ------------------------------
>
> Key: CASSANDRA-9625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9625
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3
> Reporter: Eric Evans
> Assignee: T Jake Luciani
> Attachments: metrics.yaml, thread-dump.log
>
>
> When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops
> working. The usual startup is logged, and one batch of samples is sent, but
> the reporting interval comes and goes, and no other samples are ever sent.
> The logs are free from errors.
> Frustratingly, metrics reporting works in our smaller (staging) environment
> on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not
> on a 3 node (otherwise identical) staging cluster (maybe it takes a certain
> level of concurrency?).
> Attached is a thread dump, and our metrics.yaml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
