[
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256326#comment-14256326
]
Ariel Weisberg edited comment on CASSANDRA-8457 at 12/22/14 11:48 PM:
----------------------------------------------------------------------
Repeating these benchmarks, trunk can be slower as or more often than it is
faster. It's one of those peak performance vs. average and worst case sort of
decisions.
This is odd. Some measurements running trunk. Soft interrupt hotspot on one
CPU. Don't see why that should happen. Interrupt hotspots are usually related
to not enough TCP streams and there should be several in this workload. Kind of
looks like someone somewhere pinned interrupts to core 0.
{noformat}
11:20:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:20:05 PM all 11.19 0.00 9.82 0.00 0.00 0.98 1.27
0.00 76.74
11:20:05 PM 0 18.97 0.00 12.07 0.00 0.00 32.76 1.72
0.00 34.48
11:20:05 PM 1 6.12 0.00 10.20 0.00 0.00 0.00 4.08
0.00 79.59
11:20:05 PM 2 7.55 0.00 11.32 0.00 0.00 0.00 3.77
0.00 77.36
11:20:05 PM 3 11.11 0.00 9.26 0.00 0.00 0.00 1.85
0.00 77.78
11:20:05 PM 4 17.86 0.00 5.36 0.00 0.00 0.00 0.00
0.00 76.79
11:20:05 PM 5 10.71 0.00 8.93 0.00 0.00 0.00 0.00
0.00 80.36
11:20:05 PM 6 10.17 0.00 10.17 0.00 0.00 0.00 1.69
0.00 77.97
11:20:05 PM 7 10.00 0.00 11.67 0.00 0.00 0.00 1.67
0.00 76.67
11:20:05 PM 8 13.56 0.00 6.78 0.00 0.00 0.00 1.69
0.00 77.97
11:20:05 PM 9 12.50 0.00 10.94 0.00 0.00 0.00 1.56
0.00 75.00
11:20:05 PM 10 15.15 0.00 9.09 0.00 0.00 0.00 1.52
0.00 74.24
11:20:05 PM 11 7.94 0.00 12.70 0.00 0.00 0.00 1.59
0.00 77.78
11:20:05 PM 12 9.52 0.00 9.52 0.00 0.00 0.00 1.59
0.00 79.37
11:20:05 PM 13 9.09 0.00 12.12 0.00 0.00 0.00 0.00
0.00 78.79
11:20:05 PM 14 11.76 0.00 10.29 0.00 0.00 0.00 1.47
0.00 76.47
11:20:05 PM 15 9.23 0.00 9.23 0.00 0.00 0.00 1.54
0.00 80.00
11:20:05 PM 16 10.29 0.00 14.71 0.00 0.00 0.00 1.47
0.00 73.53
11:20:05 PM 17 10.45 0.00 11.94 0.00 0.00 0.00 0.00
0.00 77.61
11:20:05 PM 18 11.94 0.00 8.96 0.00 0.00 0.00 1.49
0.00 77.61
11:20:05 PM 19 12.68 0.00 12.68 0.00 0.00 0.00 1.41
0.00 73.24
11:20:05 PM 20 13.24 0.00 8.82 0.00 0.00 0.00 0.00
0.00 77.94
11:20:05 PM 21 5.80 0.00 15.94 0.00 0.00 0.00 0.00
0.00 78.26
11:20:05 PM 22 13.89 0.00 9.72 0.00 0.00 0.00 1.39
0.00 75.00
11:20:05 PM 23 9.38 0.00 9.38 0.00 0.00 0.00 0.00
0.00 81.25
11:20:05 PM 24 7.81 0.00 9.38 0.00 0.00 0.00 1.56
0.00 81.25
11:20:05 PM 25 8.96 0.00 8.96 0.00 0.00 0.00 1.49
0.00 80.60
11:20:05 PM 26 11.59 0.00 10.14 0.00 0.00 0.00 0.00
0.00 78.26
11:20:05 PM 27 13.04 0.00 7.25 0.00 0.00 0.00 1.45
0.00 78.26
11:20:05 PM 28 7.81 0.00 7.81 0.00 0.00 0.00 0.00
0.00 84.38
11:20:05 PM 29 11.76 0.00 8.82 0.00 0.00 0.00 0.00
0.00 79.41
11:20:05 PM 30 11.43 0.00 10.00 0.00 0.00 0.00 1.43
0.00 77.14
11:20:05 PM 31 12.50 0.00 4.69 0.00 0.00 0.00 0.00
0.00 82.81
{noformat}
Modified mpstat output is similar. perf stat doesn't have access to performance
counters in these instances. I'll have to see if I can get instances that do
that. I have a flight recording of each, but not a flight recording of great
performance on trunk.
was (Author: aweisberg):
Repeating these benchmarks, trunk can be slower as or more often then it is
faster. It's one of those peak performance vs. average and worst case sort of
decisions.
This is odd. Some measurements running trunk. Soft interrupt hotspot on one
CPU. Don't see why that should happen. Interrupt hotspots are usually related
to not enough TCP streams and there should be several in this workload. Kind of
looks like someone somewhere pinned interrupts to core 0.
{noformat}
11:20:04 PM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:20:05 PM all 11.19 0.00 9.82 0.00 0.00 0.98 1.27
0.00 76.74
11:20:05 PM 0 18.97 0.00 12.07 0.00 0.00 32.76 1.72
0.00 34.48
11:20:05 PM 1 6.12 0.00 10.20 0.00 0.00 0.00 4.08
0.00 79.59
11:20:05 PM 2 7.55 0.00 11.32 0.00 0.00 0.00 3.77
0.00 77.36
11:20:05 PM 3 11.11 0.00 9.26 0.00 0.00 0.00 1.85
0.00 77.78
11:20:05 PM 4 17.86 0.00 5.36 0.00 0.00 0.00 0.00
0.00 76.79
11:20:05 PM 5 10.71 0.00 8.93 0.00 0.00 0.00 0.00
0.00 80.36
11:20:05 PM 6 10.17 0.00 10.17 0.00 0.00 0.00 1.69
0.00 77.97
11:20:05 PM 7 10.00 0.00 11.67 0.00 0.00 0.00 1.67
0.00 76.67
11:20:05 PM 8 13.56 0.00 6.78 0.00 0.00 0.00 1.69
0.00 77.97
11:20:05 PM 9 12.50 0.00 10.94 0.00 0.00 0.00 1.56
0.00 75.00
11:20:05 PM 10 15.15 0.00 9.09 0.00 0.00 0.00 1.52
0.00 74.24
11:20:05 PM 11 7.94 0.00 12.70 0.00 0.00 0.00 1.59
0.00 77.78
11:20:05 PM 12 9.52 0.00 9.52 0.00 0.00 0.00 1.59
0.00 79.37
11:20:05 PM 13 9.09 0.00 12.12 0.00 0.00 0.00 0.00
0.00 78.79
11:20:05 PM 14 11.76 0.00 10.29 0.00 0.00 0.00 1.47
0.00 76.47
11:20:05 PM 15 9.23 0.00 9.23 0.00 0.00 0.00 1.54
0.00 80.00
11:20:05 PM 16 10.29 0.00 14.71 0.00 0.00 0.00 1.47
0.00 73.53
11:20:05 PM 17 10.45 0.00 11.94 0.00 0.00 0.00 0.00
0.00 77.61
11:20:05 PM 18 11.94 0.00 8.96 0.00 0.00 0.00 1.49
0.00 77.61
11:20:05 PM 19 12.68 0.00 12.68 0.00 0.00 0.00 1.41
0.00 73.24
11:20:05 PM 20 13.24 0.00 8.82 0.00 0.00 0.00 0.00
0.00 77.94
11:20:05 PM 21 5.80 0.00 15.94 0.00 0.00 0.00 0.00
0.00 78.26
11:20:05 PM 22 13.89 0.00 9.72 0.00 0.00 0.00 1.39
0.00 75.00
11:20:05 PM 23 9.38 0.00 9.38 0.00 0.00 0.00 0.00
0.00 81.25
11:20:05 PM 24 7.81 0.00 9.38 0.00 0.00 0.00 1.56
0.00 81.25
11:20:05 PM 25 8.96 0.00 8.96 0.00 0.00 0.00 1.49
0.00 80.60
11:20:05 PM 26 11.59 0.00 10.14 0.00 0.00 0.00 0.00
0.00 78.26
11:20:05 PM 27 13.04 0.00 7.25 0.00 0.00 0.00 1.45
0.00 78.26
11:20:05 PM 28 7.81 0.00 7.81 0.00 0.00 0.00 0.00
0.00 84.38
11:20:05 PM 29 11.76 0.00 8.82 0.00 0.00 0.00 0.00
0.00 79.41
11:20:05 PM 30 11.43 0.00 10.00 0.00 0.00 0.00 1.43
0.00 77.14
11:20:05 PM 31 12.50 0.00 4.69 0.00 0.00 0.00 0.00
0.00 82.81
{noformat}
Modified mpstat output is similar. perf stat doesn't have access to performance
counters in these instances. I'll have to see if I can get instances that do
that. I have a flight recording of each, but not a flight recording of great
performance on trunk.
> nio MessagingService
> --------------------
>
> Key: CASSANDRA-8457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Ariel Weisberg
> Labels: performance
> Fix For: 3.0
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big
> contributor to context switching, especially for larger clusters. Let's look
> at switching to nio, possibly via Netty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
