[
https://issues.apache.org/jira/browse/CASSANDRA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064255#comment-15064255
]
Robert Stupp commented on CASSANDRA-9923:
-----------------------------------------
bq. Mainly stress is running low on heap.
Makes sense to me.
Maybe just add a "generic" {{-Jfoo}} option to pass JVM arguments to stress.
WDYT?
> stress write and counter_write hangs
> ------------------------------------
>
> Key: CASSANDRA-9923
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9923
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Robert Stupp
> Assignee: T Jake Luciani
> Priority: Minor
> Labels: stress
>
> (Sorry for the vague description)
> I tried some cstar tests against counter columns. But all these tests against
> 2.1 and 2.2 ended (hang) during with the following output:
> {noformat}
> Created keyspaces. Sleeping 3s for propagation.
> Sleeping 2s...
> Warming up COUNTER_WRITE with 150000 iterations...
> Running COUNTER_WRITE with 300 threads for 15000000 iteration
> type, total ops, op/s, pk/s, row/s, mean, med, .95,
> .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms,
> sdv ms, mb
> total, 98073, 98054, 98054, 98054, 3.1, 1.7, 8.9,
> 23.2, 89.9, 107.7, 1.0, 0.00000, 0, 0, 0, 0,
> 0, 0
> total, 188586, 72492, 72492, 72492, 4.1, 1.5, 10.0,
> 61.4, 202.8, 214.7, 2.2, 0.13101, 0, 3, 564, 564,
> 6, 3447
> total, 363880, 137986, 137986, 137986, 2.2, 1.4, 4.1,
> 9.6, 207.1, 253.3, 3.5, 0.18684, 0, 0, 0, 0,
> 0, 0
> total, 460122, 105062, 105062, 105062, 2.8, 1.4, 4.6,
> 14.7, 225.6, 236.2, 4.4, 0.13969, 0, 1, 214, 214,
> 0, 1199
> total, 600625, 111453, 111453, 111453, 2.7, 1.4, 3.8,
> 10.4, 231.5, 241.6, 5.7, 0.11366, 0, 2, 442, 442,
> 1, 2389
> total, 745680, 149583, 149583, 149583, 2.0, 1.4, 3.6,
> 6.7, 155.8, 159.7, 6.7, 0.11318, 0, 0, 0, 0,
> 0, 0
> total, 828453, 63632, 63632, 63632, 4.7, 1.4, 4.8,
> 261.9, 274.5, 279.3, 8.0, 0.12645, 0, 3, 782, 782,
> 1, 3542
> total, 1009560, 172429, 172429, 172429, 1.7, 1.4, 3.7,
> 6.1, 16.2, 29.7, 9.0, 0.11629, 0, 0, 0, 0,
> 0, 0
> total, 1062409, 53860, 53860, 53860, 5.5, 1.3, 8.6,
> 270.3, 293.4, 324.3, 10.0, 0.12738, 0, 2, 542, 542,
> 7, 2354
> total, 1186672, 96540, 96540, 96540, 3.1, 1.5, 5.9,
> 14.5, 266.4, 277.6, 11.3, 0.11451, 0, 1, 260, 260,
> 0, 1183
> {noformat}
> ...
> {noformat}
> total, 4977251, 238, 238, 238, 0.7, 0.6, 0.7,
> 1.3, 3.4, 158.5, 352.3, 0.11749, 0, 0, 0, 0,
> 0, 0
> total, 4979839, 214, 214, 214, 0.6, 0.6, 0.7,
> 1.3, 2.5, 2.8, 364.4, 0.11761, 0, 0, 0, 0,
> 0, 0
> total, 4981729, 191, 191, 191, 0.6, 0.6, 0.7,
> 1.3, 3.2, 3.3, 374.3, 0.11774, 0, 0, 0, 0,
> 0, 0
> total, 4983362, 167, 167, 167, 0.8, 0.7, 1.8,
> 2.7, 3.9, 5.8, 384.0, 0.11787, 0, 0, 0, 0,
> 0, 0
> total, 4985171, 153, 153, 153, 0.7, 0.6, 1.2,
> 1.4, 2.0, 3.3, 395.9, 0.11799, 0, 0, 0, 0,
> 0, 0
> total, 4986684, 137, 137, 137, 0.7, 0.6, 0.8,
> 1.3, 2.0, 2.0, 406.9, 0.11812, 0, 0, 0, 0,
> 0, 0
> total, 4988410, 121, 121, 121, 0.7, 0.7, 0.8,
> 1.3, 2.0, 2.8, 421.1, 0.11824, 0, 0, 0, 0,
> 0, 0
> total, 4990216, 99, 99, 99, 0.7, 0.7, 0.8,
> 1.4, 2.6, 2.8, 439.5, 0.11836, 0, 0, 0, 0,
> 0, 0
> total, 4991765, 81, 81, 81, 0.8, 0.7, 0.8,
> 1.4, 30.1, 81.6, 458.7, 0.11848, 0, 1, 159, 159,
> 0, 1179
> total, 4993731, 67, 67, 67, 0.7, 0.7, 0.8,
> 1.4, 3.2, 3.2, 488.1, 0.11860, 0, 0, 0, 0,
> 0, 0
> total, 4996565, 45, 45, 45, 0.9, 0.7, 0.9,
> 1.5, 84.7, 218.3, 551.5, 0.11872, 0, 1, 248, 248,
> 0, 1180
> java.lang.RuntimeException: Timed out waiting for a timer thread - seems one
> got stuck. Check GC/Heap size
> at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98)
> at
> org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
> at
> org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:37)
> at
> org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I don't know whether it's a stress or a counter issue - but wanted to record
> it for further investigation.
> [Console log
> file|http://cstar.datastax.com/tests/artifacts/6cd54260-35d6-11e5-a6ff-42010af0688f/console]
> cstar test steps:
> # counter_write n=15M -pop dist=gaussian\(1..5M\) -rate threads=300
> # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=300
> # counter_write n=15M -pop dist=gaussian\(1..500k\) -rate threads=50
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
