Hi, we use c++ thrift gateway to connect to hbase, but thrift server died frequently without any error log. below is the GC info, showing FGC increase too fast, i am not sure if this is the reasion lead thrift server down.
hadoop@hadoop3 ~]$ [hadoop@hadoop3 ~]$ jstat -gcutil 20267 4000 1000 S0 S1 E O P YGC YGCT FGC FGCT GCT 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 46.03 0.00 50.94 0 0.000 0 0.000 0.000 0.00 0.00 48.03 0.00 51.07 0 0.000 0 0.000 0.000 100.00 0.00 30.20 18.74 54.44 2 0.218 0 0.000 0.218 99.42 0.00 90.02 39.26 54.44 4 0.375 0 0.000 0.375 97.89 0.00 57.24 34.73 54.45 6 0.594 3 0.064 0.658 100.00 0.00 43.69 71.95 54.46 8 0.836 5 0.091 0.926 100.00 0.00 55.37 46.22 54.47 10 1.057 8 0.121 1.178 0.00 100.00 60.28 58.43 54.47 11 1.115 10 0.191 1.306 95.86 100.00 100.00 73.65 54.47 13 1.132 12 0.217 1.349 0.00 96.78 45.63 59.59 54.47 15 1.469 17 0.328 1.796 26.74 100.00 100.00 84.32 54.47 17 1.481 20 0.350 1.832 0.00 94.94 57.65 45.70 54.47 19 1.889 24 0.389 2.278 0.00 100.00 44.40 54.67 54.47 21 1.998 24 0.389 2.387 96.60 0.00 20.06 51.55 54.47 24 2.333 26 0.407 2.740 0.00 28.10 27.94 51.55 54.47 25 2.346 26 0.407 2.752 100.00 0.00 11.02 97.80 54.47 28 2.832 27 0.418 3.251 0.00 100.00 27.07 53.35 54.48 29 2.858 28 0.425 3.283 31.83 0.00 74.97 53.35 54.49 30 2.868 28 0.425 3.293 0.00 100.00 71.88 60.00 54.49 33 3.288 30 0.455 3.743 0.00 100.00 60.57 52.84 54.49 35 3.415 32 0.473 3.888 0.00 23.97 24.01 52.84 54.50 37 3.445 32 0.473 3.918 100.00 0.00 41.60 76.40 54.50 40 3.985 35 0.517 4.501 29.46 0.00 28.26 57.41 54.50 42 4.014 38 0.531 4.545 0.00 25.35 19.85 57.41 54.51 43 4.024 38 0.531 4.555 100.00 0.00 96.31 60.64 54.51 44 4.078 38 0.531 4.610 97.74 0.00 7.80 61.27 54.51 48 4.657 41 0.565 5.222 0.00 28.38 82.01 60.00 54.51 49 4.668 42 0.569 5.237 0.00 87.77 72.09 60.00 54.52 51 4.705 42 0.569 5.274 100.00 0.00 92.48 57.50 54.52 54 5.312 44 0.590 5.901 97.26 0.00 15.28 64.32 54.53 56 5.394 45 0.602 5.996 97.26 0.00 49.10 60.00 54.55 56 5.394 46 0.606 6.000 0.00 32.01 76.99 60.00 54.55 57 5.404 46 0.606 6.010 30.77 0.00 86.23 60.00 54.56 58 5.419 46 0.606 6.025 100.00 0.00 100.00 80.58 54.56 61 5.683 48 0.619 6.302 0.00 100.00 19.81 58.01 54.54 62 5.883 49 1.799 7.683 0.00 100.00 89.36 57.98 54.55 62 5.883 50 1.811 7.694 85.81 0.00 50.41 62.83 54.55 63 5.927 52 1.899 7.826 0.00 22.57 39.94 62.83 54.55 64 5.940 53 1.966 7.905 0.00 100.00 10.11 88.28 54.55 66 6.664 57 2.104 8.768 0.00 100.00 94.28 62.62 54.55 66 6.664 60 2.223 8.887 0.00 0.00 60.12 66.92 54.55 67 6.664 63 3.666 10.330 0.00 0.00 32.76 67.36 54.55 68 6.664 65 4.566 11.230 100.00 0.00 34.48 82.35 54.55 69 6.862 68 4.597 11.459 100.00 0.00 6.01 67.09 54.56 71 7.404 70 4.767 12.171 100.00 0.00 99.30 67.09 54.57 71 7.404 72 4.862 12.266 0.00 0.00 33.03 71.19 54.57 72 7.404 74 6.254 13.658 0.00 0.00 100.00 71.09 54.57 73 7.404 78 6.360 13.764 0.00 0.00 86.48 71.53 54.57 73 7.404 80 7.324 14.728 100.00 0.00 66.19 62.59 54.57 75 8.115 83 7.377 15.492 0.00 0.00 25.52 74.86 54.57 76 8.115 86 9.035 17.150 0.00 0.00 99.99 74.86 54.57 77 8.115 88 9.043 17.159 0.00 0.00 44.03 75.29 54.60 77 8.115 89 10.093 18.209 0.00 0.00 99.41 75.29 54.60 78 8.115 92 10.158 18.273 0.00 0.00 68.52 75.70 54.60 78 8.115 94 11.721 19.836 0.00 100.00 39.13 59.36 54.60 80 8.452 96 12.812 21.264 0.00 100.00 99.73 59.36 54.60 81 8.452 99 13.068 21.520 0.00 0.00 84.69 78.90 54.60 81 8.452 101 14.952 23.404 0.00 0.00 37.23 79.38 54.60 82 8.452 103 16.762 25.214 0.00 0.00 99.09 79.32 54.60 83 8.452 107 16.878 25.330 0.00 0.00 83.16 79.76 54.60 83 8.452 109 18.152 26.604 100.00 0.00 100.00 38.22 54.60 85 8.452 110 19.001 27.453 0.00 100.00 61.46 79.20 54.61 86 9.099 114 19.127 28.226 0.00 0.00 12.11 83.37 54.61 87 9.099 116 20.857 29.955 0.00 0.00 99.54 83.37 54.61 88 9.099 119 20.923 30.022 0.00 0.00 50.94 83.69 54.61 88 9.099 121 22.314 31.413 0.00 0.00 16.19 84.32 54.61 89 9.099 123 23.829 32.928 100.00 0.00 13.78 31.72 54.61 90 9.469 125 24.060 33.529 0.00 100.00 100.00 65.48 54.61 92 9.810 128 24.194 34.004 0.00 0.00 63.34 87.11 54.61 92 9.810 130 26.038 35.848 0.00 0.00 10.55 87.54 54.61 93 9.810 134 27.917 37.727 0.00 0.00 52.07 87.54 54.61 93 9.810 137 27.980 37.790 0.00 0.00 99.36 87.53 54.61 94 9.810 140 28.077 37.887 0.00 0.00 38.44 88.15 54.61 94 9.810 142 29.227 39.037 0.00 0.00 100.00 87.96 54.61 95 9.810 145 29.295 39.105 100.00 0.00 100.00 28.91 54.61 96 9.810 145 29.725 39.535 0.00 100.00 99.70 72.52 54.61 98 10.546 149 29.999 40.545 0.00 0.00 38.91 91.34 54.61 98 10.546 150 32.131 42.677 0.00 0.00 89.32 91.24 54.61 98 10.546 153 32.231 42.777 0.00 0.00 36.60 91.86 54.61 99 10.546 156 34.180 44.726 0.00 0.00 92.53 91.66 54.61 99 10.546 158 34.267 44.813 0.00 0.00 33.04 92.20 54.61 100 10.546 160 35.946 46.492 0.00 0.00 76.30 92.10 54.61 100 10.546 164 36.065 46.612 0.00 0.00 40.93 2.50 54.60 101 10.546 167 36.733 47.280 100.00 100.00 100.00 48.62 54.60 103 10.976 169 36.827 47.803 0.00 0.00 3.34 95.36 54.60 104 11.327 171 38.923 50.250 0.00 0.00 71.02 95.26 54.60 104 11.327 174 38.983 50.310 0.00 0.00 0.00 95.75 54.60 105 11.327 177 41.010 52.338 0.00 0.00 91.79 95.69 54.60 105 11.327 179 41.072 52.399 0.00 0.00 43.84 96.18 54.60 106 11.327 182 43.009 54.336 0.00 0.00 99.86 96.13 54.60 107 11.327 185 43.089 54.416 0.00 0.00 96.05 4.11 54.60 107 11.327 189 44.622 55.949 0.00 100.00 100.00 57.33 54.60 110 11.989 192 44.825 56.814 0.00 0.00 53.79 95.05 54.60 110 11.989 192 47.075 59.064 0.00 0.00 6.69 99.66 54.60 111 11.989 196 49.055 61.044 0.00 0.00 89.93 99.61 54.60 111 11.989 199 49.122 61.111 0.00 0.00 56.25 100.00 54.61 112 11.989 202 50.415 62.405 0.00 99.82 100.00 100.00 54.61 113 11.990 205 50.507 62.497 0.00 0.00 79.16 100.00 54.61 113 11.990 208 51.909 63.898 0.00 0.00 53.99 12.46 54.61 114 11.990 210 52.621 64.611 0.00 100.00 78.40 79.09 54.61 116 12.709 212 52.659 65.368 0.00 0.00 31.41 100.00 54.62 117 12.709 215 54.945 67.655 0.00 69.65 100.00 99.90 54.62 117 12.709 217 55.054 67.763 0.00 0.00 72.11 100.00 54.62 118 12.709 220 56.565 69.274 0.00 0.00 30.67 100.00 54.62 119 12.709 225 58.116 70.826 0.00 0.00 88.96 100.00 54.62 119 12.709 227 58.284 70.994 0.00 97.84 100.00 100.00 54.63 120 12.710 232 58.599 71.308 0.00 0.00 46.02 100.00 54.63 120 12.710 234 60.797 73.507 100.00 0.00 20.40 60.50 54.63 121 12.978 237 60.983 73.961 100.00 100.00 100.00 61.54 54.63 123 13.463 240 61.065 74.528 100.00 0.00 100.00 83.72 54.63 124 13.837 243 61.192 7 hadoop@hadoop3 conf]$ jps 21698 Jstat 20267 ThriftServer 3405 QuorumPeerMain 21407 Jstat 31825 HRegionServer 3307 DFSZKFailoverController 3668 DataNode 8479 NameNode 21879 Jps [hadoop@hadoop3 conf]$ jmap -heap 20267 Attaching to process ID 20267, please wait... Debugger attached successfully. Server compiler detected. JVM version is 22.1-b02 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 1048576000 (1000.0MB) NewSize = 1310720 (1.25MB) MaxNewSize = 349503488 (333.3125MB) OldSize = 5439488 (5.1875MB) NewRatio = 2 SurvivorRatio = 8 PermSize = 21757952 (20.75MB) MaxPermSize = 85983232 (82.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 314572800 (300.0MB) used = 104893712 (100.03443908691406MB) free = 209679088 (199.96556091308594MB) 33.34481302897135% used Eden Space: capacity = 279642112 (266.6875MB) used = 69963024 (66.72193908691406MB) free = 209679088 (199.96556091308594MB) 25.018772565986055% used >From Space: capacity = 34930688 (33.3125MB) used = 34930688 (33.3125MB) free = 0 (0.0MB) 100.0% used To Space: capacity = 34930688 (33.3125MB) used = 0 (0.0MB) free = 34930688 (33.3125MB) 0.0% used concurrent mark-sweep generation: capacity = 699072512 (666.6875MB) used = 405508344 (386.72289276123047MB) free = 293564168 (279.96460723876953MB) 58.00662120727184% used Perm Generation: capacity = 21757952 (20.75MB) used = 11866416 (11.316696166992188MB) free = 9891536 (9.433303833007812MB) 54.53829478068524% used how to change heapsize of thrift servr? Best R beatls
