[
https://issues.apache.org/jira/browse/CASSANDRA-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shawn Kumar updated CASSANDRA-8008:
-----------------------------------
Description:
I've been using cstar_perf to test a performance scenario and was able to
reproduce this error on stock 2.1.0 while carrying out large stress writes (50M
keys):
{noformat}
java.lang.RuntimeException: Timed out waiting for a timer thread - seems one
got stuck
at org.apache.cassandra.stress.util.Timing.snap(Timing.java:83)
at org.apache.cassandra.stress.util.Timing.snap(Timing.java:118)
at
org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
at
org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:42)
at
org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
at java.lang.Thread.run(Thread.java:745)
{noformat}
It looks similar to CASSANDRA-6943, but that should have fixed it, and I
haven't been able to consistently replicate this with other runs. This
particular run was stress writing/reading about 300M keys, and is an early
attempt at carrying out a test of this size so perhaps it only manifests with
larger tests.
was:
I've been using cstar_perf to test cassandra with different gc's and came
across this error on one run which effectively stopped the test:
java.lang.RuntimeException: Timed out waiting for a timer thread - seems one
got stuck at org.apache.cassandra.stress.util.Timing.snap(Timing.java:83)
It looks similar to CASSANDRA-6943, but that should have fixed it, and I
haven't been able to consistently replicate this with other runs. This
particular run was stress writing/reading about 300M keys, and is an early
attempt at carrying out a test of this size so perhaps it only manifests with
larger tests.
The modifications from stock 2.1.0 were changes to heap size and usage of g1gc,
as well as using offheap_objects. I have attached thread dumps from the nodes
in question, hopefully they capture the broken state. I am continuing to test
this, and will see if I can reproduce this again.
> "Timed out waiting for timer thread" on large stress command
> ------------------------------------------------------------
>
> Key: CASSANDRA-8008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8008
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Shawn Kumar
> Attachments: node1.log, node2.log
>
>
> I've been using cstar_perf to test a performance scenario and was able to
> reproduce this error on stock 2.1.0 while carrying out large stress writes
> (50M keys):
> {noformat}
> java.lang.RuntimeException: Timed out waiting for a timer thread - seems one
> got stuck
> at org.apache.cassandra.stress.util.Timing.snap(Timing.java:83)
> at org.apache.cassandra.stress.util.Timing.snap(Timing.java:118)
> at
> org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
> at
> org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:42)
> at
> org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> It looks similar to CASSANDRA-6943, but that should have fixed it, and I
> haven't been able to consistently replicate this with other runs. This
> particular run was stress writing/reading about 300M keys, and is an early
> attempt at carrying out a test of this size so perhaps it only manifests with
> larger tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
