[jira] [Commented] (CASSANDRA-6410) gossip memory usage improvement
[ https://issues.apache.org/jira/browse/CASSANDRA-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838372#comment-13838372 ] Quentin Conner commented on CASSANDRA-6410: --- Chris, the Heap memory usage numbers are from a single node. No aggregation across the cluster. gossip memory usage improvement --- Key: CASSANDRA-6410 URL: https://issues.apache.org/jira/browse/CASSANDRA-6410 Project: Cassandra Issue Type: Improvement Reporter: Quentin Conner Assignee: Jonathan Ellis Priority: Minor Fix For: 2.0.4 Attachments: 6410-EnumMap.txt, gossip-intern.txt It looks to me that any given node will need ~2 MB of Java VM heap for each other node in the ring. This was observed with num_tokens=512 but still seems excessive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6410) gossip memory usage improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837088#comment-13837088
]
Quentin Conner commented on CASSANDRA-6410:
---
After starting up a 500 node cluster (num_tokens=512), I tested the
6410-Enum.txt patch.
I captured java heap usage via jmxterm before and after the patch. There
seemed to be slightly less memory used. CPU used also appears to be less after
reaching steady state.
*BEFORE*
{code}
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 483368936;
};
null
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 218900120;
};
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 280581144;
};
null
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 219137176;
};
{code}
~209 MB before the patch @ 500 nodes
{code}
09:58:01 PM CPU %user %nice %system %iowait%steal %idle
09:59:01 PM all 1.11 0.00 0.25 0.05 0.02 98.57
10:00:01 PM all 0.99 0.00 0.40 0.02 0.02 98.57
10:01:01 PM all 1.02 0.00 0.40 0.05 0.02 98.51
10:02:01 PM all 1.11 0.00 0.40 0.02 0.02 98.46
10:03:01 PM all 1.21 0.00 0.40 0.03 0.00 98.36
10:04:01 PM all 1.37 0.00 0.45 0.05 0.02 98.11
10:05:01 PM all 1.12 0.00 0.40 0.03 0.02 98.42
10:06:01 PM all 1.06 0.00 0.37 0.03 0.02 98.53
10:07:01 PM all 1.27 0.00 0.34 0.05 0.02 98.32
10:08:01 PM all 1.26 0.00 0.32 0.07 0.00 98.36
10:09:02 PM all 1.42 0.00 0.27 0.03 0.02 98.26
10:10:01 PM all 0.92 0.00 0.32 0.07 0.02 98.67
10:11:01 PM all 1.28 0.00 0.30 0.03 0.02 98.37
10:12:01 PM all 1.09 0.00 0.25 0.02 0.02 98.62
10:13:01 PM all 1.14 0.00 0.35 0.02 0.02 98.47
10:14:01 PM all 12.95 0.00 1.81 0.03 0.03 85.16
10:15:01 PM all 1.38 0.00 0.40 0.07 0.02 98.13
10:16:01 PM all 52.16 0.00 3.20 0.05 0.03 44.55
10:17:01 PM all 5.22 0.00 0.52 0.10 0.02 94.14
10:18:01 PM all 9.90 0.00 0.45 0.17 0.02 89.46
10:19:01 PM all 9.50 0.00 0.30 0.12 0.02 90.06
10:20:01 PM all 5.32 0.00 0.33 0.13 0.02 94.19
10:21:01 PM all 6.30 0.00 0.42 0.13 0.00 93.15
10:22:01 PM all 23.79 0.00 2.30 0.10 0.03 73.77
10:23:01 PM all 9.00 0.00 0.49 0.10 0.03 90.38
10:24:01 PM all 3.56 0.00 0.52 0.10 0.00 95.82
10:25:01 PM all 3.12 0.00 0.45 0.10 0.02 96.31
10:26:01 PM all 2.15 0.00 0.39 0.08 0.02 97.37
10:27:01 PM all 2.28 0.00 0.42 0.05 0.02 97.23
{code}
*AFTER*
{code}
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 230930624;
};
null
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 199311088;
};
get -b java.lang:type=Memory HeapMemoryUsage
run -b java.lang:type=Memory gc
get -b java.lang:type=Memory HeapMemoryUsage
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 209453840;
};
null
HeapMemoryUsage = {
committed = 2063597568;
init = 2147483648;
max = 2063597568;
used = 205107928;
};
{code}
~196 MB after the patch @ 500 nodes
{code}
11:14:01 PM CPU %user %nice %system %iowait%steal %idle
11:15:01 PM all 9.39 0.00 0.79 0.12 0.02 89.69
11:16:01 PM all 1.09 0.00 0.75 0.15 0.02 97.99
11:17:01 PM all 0.59 0.00 0.22 0.10 0.00 99.10
11:18:01 PM all 0.74 0.00 0.17 0.08 0.02 98.99
11:19:01 PM all 11.01 0.00 2.29 0.15 0.02 86.53
11:20:01 PM all 1.11 0.00 0.34 0.12 0.03 98.41
11:21:01 PM all 2.55 0.00 0.67 0.22 0.02 96.54
11:22:01 PM all 24.31 0.00 4.70 0.07 0.03 70.89
11:23:01 PM all 25.78 0.00 2.58 0.32 0.02 71.31
11:24:01 PM all 11.60 0.00 2.50 0.07 0.03
[jira] [Commented] (CASSANDRA-6410) gossip memory usage improvement
[ https://issues.apache.org/jira/browse/CASSANDRA-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837166#comment-13837166 ] Chris Burroughs commented on CASSANDRA-6410: Are those HeapMemoryUsage and such means or other aggregates across the 500 nodes? gossip memory usage improvement --- Key: CASSANDRA-6410 URL: https://issues.apache.org/jira/browse/CASSANDRA-6410 Project: Cassandra Issue Type: Improvement Reporter: Quentin Conner Assignee: Jonathan Ellis Priority: Minor Fix For: 1.2.13, 2.0.4 Attachments: 6410-EnumMap.txt, gossip-intern.txt It looks to me that any given node will need ~2 MB of Java VM heap for each other node in the ring. This was observed with num_tokens=512 but still seems excessive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6410) gossip memory usage improvement
[ https://issues.apache.org/jira/browse/CASSANDRA-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833934#comment-13833934 ] Quentin Conner commented on CASSANDRA-6410: --- +1 The gossip-intern.txt patch worked well, reducing server heap usage (casually measured) from 2 GB to 512 MB, when 500 node cluster was up. gossip memory usage improvement --- Key: CASSANDRA-6410 URL: https://issues.apache.org/jira/browse/CASSANDRA-6410 Project: Cassandra Issue Type: Bug Reporter: Quentin Conner Attachments: 6410-EnumMap.txt, gossip-intern.txt It looks to me that any given node will need ~2 MB of Java VM heap for each other node in the ring. This was observed with num_tokens=512 but still seems excessive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6410) gossip memory usage improvement
[ https://issues.apache.org/jira/browse/CASSANDRA-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833949#comment-13833949 ] Jonathan Ellis commented on CASSANDRA-6410: --- How about w/ the 2nd patch? gossip memory usage improvement --- Key: CASSANDRA-6410 URL: https://issues.apache.org/jira/browse/CASSANDRA-6410 Project: Cassandra Issue Type: Bug Reporter: Quentin Conner Attachments: 6410-EnumMap.txt, gossip-intern.txt It looks to me that any given node will need ~2 MB of Java VM heap for each other node in the ring. This was observed with num_tokens=512 but still seems excessive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6410) gossip memory usage improvement
[ https://issues.apache.org/jira/browse/CASSANDRA-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833978#comment-13833978 ] Quentin Conner commented on CASSANDRA-6410: --- I'll plan to start another large cluster next week and can test then. Let's say 3/3 for the ETA. gossip memory usage improvement --- Key: CASSANDRA-6410 URL: https://issues.apache.org/jira/browse/CASSANDRA-6410 Project: Cassandra Issue Type: Bug Reporter: Quentin Conner Attachments: 6410-EnumMap.txt, gossip-intern.txt It looks to me that any given node will need ~2 MB of Java VM heap for each other node in the ring. This was observed with num_tokens=512 but still seems excessive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6410) gossip memory usage improvement
[ https://issues.apache.org/jira/browse/CASSANDRA-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833344#comment-13833344 ] Quentin Conner commented on CASSANDRA-6410: --- Will see if we can find the responsible class with memory profiling gossip memory usage improvement --- Key: CASSANDRA-6410 URL: https://issues.apache.org/jira/browse/CASSANDRA-6410 Project: Cassandra Issue Type: Bug Reporter: Quentin Conner It looks to me that any given node will need ~2 MB of Java VM heap for each other node in the ring. This was observed with num_tokens=512 but still seems excessive. -- This message was sent by Atlassian JIRA (v6.1#6144)
