Matt Robenolt created CASSANDRA-7419:
----------------------------------------
Summary: Compactions not utilizing full compactionthroughput
Key: CASSANDRA-7419
URL: https://issues.apache.org/jira/browse/CASSANDRA-7419
Project: Cassandra
Issue Type: Improvement
Environment: Ubuntu 12.04, kernel 3.13
Reporter: Matt Robenolt
Priority: Minor
Fix For: 2.0.8
compactionthroughput has been set to 128MB/s, but we never see utilization that
high during compactions. Also while compacting does not seem to utilize a
single core of CPU either. So not sure what the limitation actually is, but
would expect to get all 128MB/s (or close to it).
Here is a sample of mpstat during a series of compactions after a nodetool
cleanup:
{code}
$ sudo mpstat -P ALL 1
Linux 3.13.0-24-generic (cass-activity-4.disqus.net) 06/19/2014
_x86_64_ (24 CPU)
04:55:10 PM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
04:55:11 PM all 1.42 7.36 3.43 0.04 0.33 0.00 0.00
0.00 87.41
04:55:11 PM 0 3.06 13.27 6.12 0.00 3.06 0.00 0.00
0.00 74.49
04:55:11 PM 1 2.00 23.00 6.00 0.00 0.00 0.00 0.00
0.00 69.00
04:55:11 PM 2 2.02 18.18 8.08 0.00 0.00 0.00 0.00
0.00 71.72
04:55:11 PM 3 1.98 15.84 4.95 0.00 0.99 0.00 0.00
0.00 76.24
04:55:11 PM 4 2.00 9.00 3.00 0.00 0.00 0.00 0.00
0.00 86.00
04:55:11 PM 5 1.02 22.45 10.20 0.00 0.00 0.00 0.00
0.00 66.33
04:55:11 PM 6 1.98 8.91 5.94 0.00 0.00 0.00 0.00
0.00 83.17
04:55:11 PM 7 1.00 8.00 3.00 0.00 1.00 0.00 0.00
0.00 87.00
04:55:11 PM 8 3.96 12.87 7.92 0.00 0.00 0.00 0.00
0.00 75.25
04:55:11 PM 9 2.02 5.05 2.02 0.00 0.00 0.00 0.00
0.00 90.91
04:55:11 PM 10 1.01 14.14 6.06 0.00 0.00 0.00 0.00
0.00 78.79
04:55:11 PM 11 2.00 2.00 2.00 1.00 0.00 0.00 0.00
0.00 93.00
04:55:11 PM 12 0.00 3.00 1.00 0.00 0.00 0.00 0.00
0.00 96.00
04:55:11 PM 13 2.00 0.00 2.00 0.00 1.00 0.00 0.00
0.00 95.00
04:55:11 PM 14 1.01 1.01 1.01 0.00 0.00 0.00 0.00
0.00 96.97
04:55:11 PM 15 3.03 4.04 2.02 0.00 1.01 0.00 0.00
0.00 89.90
04:55:11 PM 16 2.00 2.00 3.00 0.00 1.00 0.00 0.00
0.00 92.00
04:55:11 PM 17 1.00 2.00 0.00 0.00 0.00 0.00 0.00
0.00 97.00
04:55:11 PM 18 1.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 99.00
04:55:11 PM 19 1.00 6.00 5.00 0.00 1.00 0.00 0.00
0.00 87.00
04:55:11 PM 20 0.00 1.01 0.00 0.00 0.00 0.00 0.00
0.00 98.99
04:55:11 PM 21 0.00 1.01 1.01 0.00 0.00 0.00 0.00
0.00 97.98
04:55:11 PM 22 0.00 2.00 1.00 0.00 0.00 0.00 0.00
0.00 97.00
04:55:11 PM 23 0.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 98.00
04:55:11 PM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
04:55:12 PM all 0.88 7.44 3.01 0.00 0.17 0.00 0.00
0.00 88.50
04:55:12 PM 0 4.04 5.05 3.03 0.00 2.02 0.00 0.00
0.00 85.86
04:55:12 PM 1 0.99 12.87 5.94 0.00 0.99 0.00 0.00
0.00 79.21
04:55:12 PM 2 0.00 15.15 4.04 0.00 0.00 0.00 0.00
0.00 80.81
04:55:12 PM 3 0.00 17.17 8.08 0.00 0.00 0.00 0.00
0.00 74.75
04:55:12 PM 4 1.01 8.08 2.02 0.00 0.00 0.00 0.00
0.00 88.89
04:55:12 PM 5 1.00 20.00 3.00 0.00 0.00 0.00 0.00
0.00 76.00
04:55:12 PM 6 1.00 6.00 1.00 0.00 0.00 0.00 0.00
0.00 92.00
04:55:12 PM 7 1.01 10.10 2.02 0.00 0.00 0.00 0.00
0.00 86.87
04:55:12 PM 8 2.00 28.00 17.00 0.00 0.00 0.00 0.00
0.00 53.00
04:55:12 PM 9 1.00 18.00 9.00 1.00 0.00 0.00 0.00
0.00 71.00
04:55:12 PM 10 1.00 12.00 5.00 0.00 0.00 0.00 0.00
0.00 82.00
04:55:12 PM 11 1.00 3.00 1.00 0.00 0.00 0.00 0.00
0.00 95.00
04:55:12 PM 12 0.00 1.02 0.00 0.00 0.00 0.00 0.00
0.00 98.98
04:55:12 PM 13 1.00 1.00 2.00 0.00 1.00 0.00 0.00
0.00 95.00
04:55:12 PM 14 1.00 5.00 2.00 0.00 1.00 0.00 0.00
0.00 91.00
04:55:12 PM 15 0.99 2.97 0.99 0.00 0.99 0.00 0.00
0.00 94.06
04:55:12 PM 16 0.00 1.01 1.01 0.00 0.00 0.00 0.00
0.00 97.98
04:55:12 PM 17 0.00 3.00 1.00 0.00 0.00 0.00 0.00
0.00 96.00
04:55:12 PM 18 0.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 98.00
04:55:12 PM 19 2.00 0.00 2.00 0.00 0.00 0.00 0.00
0.00 96.00
04:55:12 PM 20 0.00 2.97 0.99 0.00 0.00 0.00 0.00
0.00 96.04
04:55:12 PM 21 0.00 3.03 0.00 0.00 0.00 0.00 0.00
0.00 96.97
04:55:12 PM 22 0.00 1.01 0.00 0.00 0.00 0.00 0.00
0.00 98.99
04:55:12 PM 23 0.00 1.01 0.00 0.00 0.00 0.00 0.00
0.00 98.99
04:55:12 PM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
04:55:13 PM all 1.84 7.23 3.01 0.04 0.25 0.00 0.00
0.00 87.64
04:55:13 PM 0 3.03 6.06 3.03 0.00 2.02 0.00 0.00
0.00 85.86
04:55:13 PM 1 3.03 5.05 2.02 0.00 0.00 0.00 0.00
0.00 89.90
04:55:13 PM 2 2.97 15.84 4.95 0.00 0.00 0.00 0.00
0.00 76.24
04:55:13 PM 3 2.00 22.00 5.00 0.00 0.00 0.00 0.00
0.00 71.00
04:55:13 PM 4 2.02 23.23 11.11 0.00 0.00 0.00 0.00
0.00 63.64
04:55:13 PM 5 1.98 17.82 3.96 0.99 0.99 0.00 0.00
0.00 74.26
04:55:13 PM 6 2.02 10.10 5.05 0.00 0.00 0.00 0.00
0.00 82.83
04:55:13 PM 7 2.97 2.97 0.99 0.00 0.00 0.00 0.00
0.00 93.07
04:55:13 PM 8 1.02 6.12 0.00 0.00 0.00 0.00 0.00
0.00 92.86
04:55:13 PM 9 1.00 4.00 2.00 0.00 0.00 0.00 0.00
0.00 93.00
04:55:13 PM 10 3.00 3.00 2.00 0.00 0.00 0.00 0.00
0.00 92.00
04:55:13 PM 11 1.01 4.04 2.02 0.00 0.00 0.00 0.00
0.00 92.93
04:55:13 PM 12 0.00 1.96 0.98 0.00 0.00 0.00 0.00
0.00 97.06
04:55:13 PM 13 1.01 0.00 1.01 0.00 0.00 0.00 0.00
0.00 97.98
04:55:13 PM 14 2.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 96.00
04:55:13 PM 15 0.00 0.00 1.02 0.00 0.00 0.00 0.00
0.00 98.98
04:55:13 PM 16 3.00 0.00 1.00 0.00 1.00 0.00 0.00
0.00 95.00
04:55:13 PM 17 1.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 98.00
04:55:13 PM 18 1.00 1.00 0.00 0.00 0.00 0.00 0.00
0.00 98.00
04:55:13 PM 19 2.00 1.00 1.00 0.00 1.00 0.00 0.00
0.00 95.00
04:55:13 PM 20 1.01 46.46 21.21 0.00 0.00 0.00 0.00
0.00 31.31
04:55:13 PM 21 2.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 96.00
04:55:13 PM 22 1.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 97.00
04:55:13 PM 23 2.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 96.00
04:55:13 PM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
04:55:14 PM all 0.75 7.45 3.22 0.04 0.21 0.00 0.00
0.00 88.33
04:55:14 PM 0 3.00 5.00 3.00 0.00 3.00 0.00 0.00
0.00 86.00
04:55:14 PM 1 1.01 13.13 5.05 0.00 0.00 0.00 0.00
0.00 80.81
04:55:14 PM 2 1.01 25.25 14.14 0.00 0.00 0.00 0.00
0.00 59.60
04:55:14 PM 3 2.02 13.13 3.03 0.00 0.00 0.00 0.00
0.00 81.82
04:55:14 PM 4 2.00 11.00 2.00 0.00 0.00 0.00 0.00
0.00 85.00
04:55:14 PM 5 1.00 17.00 5.00 0.00 0.00 0.00 0.00
0.00 77.00
04:55:14 PM 6 1.01 26.26 10.10 0.00 0.00 0.00 0.00
0.00 62.63
04:55:14 PM 7 1.01 4.04 2.02 0.00 0.00 0.00 0.00
0.00 92.93
04:55:14 PM 8 1.00 4.00 2.00 0.00 0.00 0.00 0.00
0.00 93.00
04:55:14 PM 9 1.01 19.19 9.09 0.00 0.00 0.00 0.00
0.00 70.71
04:55:14 PM 10 1.01 5.05 2.02 0.00 0.00 0.00 0.00
0.00 91.92
04:55:14 PM 11 1.98 5.94 1.98 0.00 0.00 0.00 0.00
0.00 90.10
04:55:14 PM 12 1.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 99.00
04:55:14 PM 13 0.00 0.00 2.00 0.00 1.00 0.00 0.00
0.00 97.00
04:55:14 PM 14 0.00 7.07 5.05 0.00 0.00 0.00 0.00
0.00 87.88
04:55:14 PM 15 0.99 0.00 0.99 0.00 0.99 0.00 0.00
0.00 97.03
04:55:14 PM 16 1.00 0.00 2.00 0.00 0.00 0.00 0.00
0.00 97.00
04:55:14 PM 17 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
04:55:14 PM 18 0.00 4.04 1.01 0.00 0.00 0.00 0.00
0.00 94.95
04:55:14 PM 19 0.00 1.00 1.00 0.00 1.00 0.00 0.00
0.00 97.00
04:55:14 PM 20 0.00 13.13 3.03 0.00 0.00 0.00 0.00
0.00 83.84
04:55:14 PM 21 0.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 98.00
04:55:14 PM 22 1.00 1.00 1.00 0.00 0.00 0.00 0.00
0.00 97.00
04:55:14 PM 23 0.00 1.98 0.99 0.00 0.00 0.00 0.00
0.00 97.03
{code}
During this time, we're seeing ~10-20MB/s on the disk which amounts to ~10%
utilization.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
