Hi all,
I have 5 nodes in my C* cluster, and each node has the same configuration
file(Cassandra-env.sh: MAX_HEAP_SIZE="32G" and HEAP_NEWSIZE="8G"), and My
Cassandra version is 2.1.1. Now I want to export all data of one table, i am
using select * from tablename, and set the bStatement.setFetchSize(10000);
When exporting the data, I check the status of the C* cluster, I found the
following:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 134.13.13.36 15.82 GB 256 ?
67aecea2-00f5-4001-ba4c-dc6138edaaa0 rack1
UN 134.13.13.37 20.25 GB 256 ?
a8c0ec24-6e48-4082-be53-32a971a4aaaa rack1
DN 134.13.13.33 18.36 GB 256 ?
d93f8201-a3a7-404a-8dd6-164ddbfeaaa7 rack1
UN 134.13.13.34 16.9 GB 256 ?
6880cf40-0b73-4bbc-a0a5-7799a983aaab rack1
UN 134.13.13.35 19.78 GB 256 ?
5658ae76-40a0-441f-908e-9d11ab2baaad rack1
The Cassandra jstat information on node 134.13.13.33 is following. I can see
that Cassandra does not use S1, and it does not do the young GC, instead, it
just do Full GC, and it do so many Full GC and so long, so C* cluster think it
is Down. The node 134.13.13.33 can only work normally after startup for a
while, after many times to do Full GC, it is Down and cannot finish the Full GC
progress and crash at the end.
[cache@cache02 caslog]$ jstat -gcutil 62335 4000 2000
S0 S1 E O P YGC YGCT FGC FGCT GCT
86.49 0.00 100.00 98.02 72.54 19 38.236 3 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.54 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.52 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.52 19 38.236 4 0.863 39.099
86.49 0.00 100.00 98.02 72.52 19 38.236 4 0.863 39.099
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 41.13 100.00 72.52 19 38.236 4 71.671 109.906
0.00 0.00 77.65 100.00 60.03 19 38.236 5 71.671 109.906
20.27 0.00 100.00 100.00 60.27 19 38.236 5 73.905 112.141
58.59 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
98.19 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 5 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
100.00 0.00 100.00 100.00 60.30 19 38.236 6 73.905 112.141
0.00 0.00 78.90 100.00 60.30 19 38.236 6 141.773 180.008
0.00 0.00 78.90 100.00 60.30 19 38.236 6 141.773 180.008
0.00 0.00 78.90 100.00 60.30 19 38.236 6 141.773 180.008
Other Cassandra jstat information of other node of this C* cluster is
following, I think it is normal because it can do young GC so many times then
do Full GC:
[cache@cache06 ~]$ jstat -gcutil 32060 4000 2000
S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 100.00 26.53 59.08 60.02 609 140.372 12 89.422 229.794
0.00 100.00 33.48 61.20 60.02 611 140.834 12 89.422 230.257
0.00 100.00 50.18 63.71 60.02 613 141.384 12 89.422 230.807
11.65 0.00 83.51 66.31 60.02 616 141.723 12 89.422 231.146
100.00 100.00 100.00 68.40 60.02 619 142.167 12 89.422 231.590
0.00 100.00 13.15 71.60 60.02 621 143.195 12 89.422 232.618
0.00 100.00 27.69 73.89 60.02 623 143.679 12 89.422 233.102
100.00 0.00 71.57 74.60 60.02 624 143.772 12 89.422 233.194
0.00 94.92 88.87 74.97 60.02 625 143.838 12 89.422 233.260
100.00 100.00 100.00 76.37 60.02 627 143.942 13 89.577 233.519
0.00 100.00 78.16 76.32 60.02 627 144.027 14 90.273 234.300
0.00 100.00 31.22 58.51 60.02 629 144.425 14 90.273 234.698
100.00 0.00 27.90 44.76 60.02 632 145.104 14 90.273 235.377
100.00 0.00 32.10 29.26 60.02 634 145.254 14 90.273 235.527
83.50 0.00 65.04 27.60 59.78 636 145.395 14 90.273 235.669
100.00 0.00 72.60 28.20 59.78 638 145.534 14 90.273 235.807
0.00 100.00 5.97 28.73 59.78 641 145.703 14 90.273 235.976
0.00 76.34 34.28 29.32 59.78 643 145.877 14 90.273 236.150
0.00 97.42 91.21 29.50 59.79 645 145.971 14 90.273 236.244
0.00 100.00 94.70 29.72 59.79 647 146.070 14 90.273 236.343
78.65 0.00 46.82 30.19 59.79 650 146.201 14 90.273 236.475
59.25 0.00 49.98 30.19 59.79 652 146.267 14 90.273 236.540
