Adding new node is really slow when you have a large load(for me, slow means several hours). So I'm interested, is there anyway to speed up the addition when adding a new node?
Best Regards, -Simon 发件人： qf zhou 发送时间： 2018-01-03 11:30 收件人： user@cassandra.apache.org 主题： Cassandra cluster add new node slowly The cluster has 3 nodes, and the data in each node is about 1.2 T. I want to add two new nodes to expand the cluster. Following the instructions from the datastax website, ie, (http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/operations/opsAddNodeToCluster.html), I try to add one node to the cluster. However, it is too slow and time cost too much. After about 24 hours, it still didn’t success. I run the command: nodetool netstats on the new node, it shows that: (tb1fullwithstate2 is a big table and 90% of the cluster data is in it. Here I use CompactionStrategy: TimeWindowCompactionStrategy). /*.*.*.3 Receiving 136 files, 328573794609 bytes total. Already received 8 files, 5774621188 bytes total tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%) received from idx:0/*.*.*.3 system_distributed/repair_history 57534/57534 bytes(100%) received from idx:0/*.*.*.3 system_distributed/parent_repair_history 507660/507660 bytes(100%) received from idx:0/*.*.*.3 tb1/tb1_device_last_state_eachday 15754096/15754096 bytes(100%) received from idx:0/*.*.*.3 mytest1/tb1_test1 8143775/8143775 bytes(100%) received from idx:0/*.*.*.3 tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%) received from idx:0/*.*.*.3 applocationinfo/weiyirong_app 2760/2760 bytes(100%) received from idx:0/*.*.*.3 tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%) received from idx:0/*.*.*.3 tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%) received from idx:0/*.*.*.3 /*.*.*.2 Receiving 136 files, 336762487360 bytes total. Already received 3 files, 5695770181 bytes total system_distributed/repair_history 31684/31684 bytes(100%) received from idx:0/*.*.*.2 tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received from idx:0/*.*.*.2 tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%) received from idx:0/*.*.*.2 tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%) received from idx:0/*.*.*.2 /*.*.*.4 Receiving 132 files, 236250553620 bytes total. Already received 10 files, 3117465128 bytes total mytest1/wordstest2 46/46 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%) received from idx:0/*.*.*.4 system_distributed/repair_history 39720/39720 bytes(100%) received from idx:0/*.*.*.4 system_distributed/parent_repair_history 452250/452250 bytes(100%) received from idx:0/*.*.*.4 mytest1/weblogs 104/104 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1_device_last_state_eachday 12670998/12670998 bytes(100%) received from idx:0/*.*.*.4 mytest1/tb1_test1 3257952/3257952 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received from idx:0/*.*.*.4 applocationinfo/weiyirong_app 3509/3509 bytes(100%) received from idx:0/*.*.*.4 tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%) received from idx:0/*.*.*.4 tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%) received from idx:0/*.*.*.4 check the log in the logs/system.log, it shows that: INFO 06:09:33 Updating topology for /*.*.*.2 INFO 06:09:33 Updating topology for /*.*.*.2 INFO 06:09:33 Node /*.*.*.3 has restarted, now UP INFO 06:09:33 InetAddress /*.*.*.3 is now UP INFO 06:09:33 Node /*.*.*.3 state jump to NORMAL INFO 06:09:33 Updating topology for /*.*.*.3 INFO 06:09:33 Updating topology for /*.*.*.3 INFO 06:09:33 Handshaking version with /*.*.*.2 INFO 06:09:33 Handshaking version with /*.*.*.3 INFO 06:09:36 Handshaking version with /*.*.*.4 INFO 06:10:03 JOINING: Starting to bootstrap... INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing streaming plan for Bootstrap INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.4 INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.4 INFO 06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints INFO 06:10:04 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints INFO 06:10:11 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints INFO 06:10:12 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints INFO 06:10:20 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints INFO 06:10:21 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints INFO 06:10:28 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 132 files(220.025GiB), sending 0 files(0.000KiB) INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.2 INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.2 INFO 06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints INFO 06:10:34 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints INFO 06:10:42 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints INFO 06:10:58 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(313.635GiB), sending 0 files(0.000KiB) INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.3 INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.3 INFO 06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints INFO 06:11:06 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints INFO 06:11:29 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(306.008GiB), sending 0 files(0.000KiB) INFO 06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints INFO 06:11:36 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints INFO 06:11:49 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints INFO 06:11:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints INFO 06:12:26 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818 INFO 06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints INFO 06:12:45 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719 INFO 06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints INFO 06:12:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87 INFO 06:24:03 G1 Young Generation GC in 242ms. G1 Eden Space: 18924699648 -> 0; G1 Old Gen: 4360876536 -> 5095617016; G1 Survivor Space: 1677721600 - INFO 06:26:06 G1 Young Generation GC in 215ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 5981077496 -> 6597610488; G1 Survivor Space: 1493172224 - INFO 06:29:04 G1 Young Generation GC in 225ms. G1 Eden Space: 18538823680 -> 0; G1 Old Gen: 7910457840 -> 8860639728; G1 Survivor Space: 1895825408 - INFO 06:33:22 G1 Young Generation GC in 214ms. G1 Eden Space: 16559112192 -> 0; G1 Old Gen: 11868321784 -> 11941600752; G1 Survivor Space: 1241513984 INFO 06:33:33 G1 Young Generation GC in 205ms. G1 Eden Space: 8069840896 -> 0; G1 Old Gen: 11941600752 -> 13683606520; G1 Survivor Space: 1895825408 INFO 06:41:01 G1 Young Generation GC in 383ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4770399736 -> 5551887344; G1 Survivor Space: 1728053248 - INFO 06:45:07 G1 Young Generation GC in 226ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8036286448 -> 8739698672; G1 Survivor Space: 1577058304 - INFO 06:47:11 G1 Young Generation GC in 203ms. G1 Eden Space: 18538823680 -> 0; G1 Survivor Space: 1191182336 -> 1778384896; INFO 06:47:22 G1 Young Generation GC in 241ms. G1 Eden Space: 8086618112 -> 0; G1 Old Gen: 9999220720 -> 11591876088; G1 Survivor Space: 1778384896 - INFO 06:49:43 G1 Young Generation GC in 201ms. G1 Eden Space: 16022241280 -> 0; G1 Old Gen: 11591876088 -> 12084766712; G1 Survivor Space: 1644167168 INFO 06:53:21 G1 Young Generation GC in 322ms. G1 Eden Space: 13237223424 -> 0; G1 Old Gen: 14755561464 -> 15445886448; G1 Survivor Space: 1459617792 INFO 06:55:34 G1 Young Generation GC in 343ms. G1 Eden Space: 16525557760 -> 0; G1 Old Gen: 2939907056 -> 3636770792; G1 Survivor Space: 1610612736 - INFO 07:09:30 Redistributing index summaries INFO 07:14:25 G1 Young Generation GC in 270ms. G1 Eden Space: 15602810880 -> 0; G1 Old Gen: 4360518136 -> 5243274224; G1 Survivor Space: 1610612736 - INFO 07:16:51 G1 Young Generation GC in 229ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6132072440 -> 6815145976; G1 Survivor Space: 1593835520 - INFO 07:19:34 G1 Young Generation GC in 215ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 8689337336 -> 9071161328; G1 Survivor Space: 1610612736 - INFO 07:23:35 G1 Young Generation GC in 293ms. G1 Eden Space: 15753805824 -> 0; G1 Old Gen: 11777605616 -> 12289649640; G1 Survivor Space: 1694498816 INFO 07:29:42 G1 Young Generation GC in 362ms. G1 Eden Space: 15955132416 -> 0; G1 Old Gen: 3742637560 -> 4410361824; G1 Survivor Space: 1644167168 - INFO 07:35:12 G1 Young Generation GC in 208ms. G1 Eden Space: 19478347776 -> 0; G1 Old Gen: 7199737832 -> 7208126432; G1 Survivor Space: 1124073472 - INFO 07:35:33 G1 Young Generation GC in 219ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 7208126432 -> 8072478184; G1 Survivor Space: 1862270976 - INFO 07:45:19 G1 Young Generation GC in 213ms. G1 Eden Space: 13153337344 -> 0; G1 Old Gen: 15418261472 -> 16123805152; G1 Survivor Space: 1174405120 INFO 07:48:08 G1 Young Generation GC in 335ms. G1 Eden Space: 18387828736 -> 0; G1 Old Gen: 3852453368 -> 4464797168; G1 Survivor Space: 1644167168 - INFO 07:48:08 G1 Young Generation GC in 209ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008 -> INFO 07:50:25 G1 Young Generation GC in 221ms. G1 Eden Space: 17498636288 -> 0; G1 Old Gen: 5981077496 -> 6488668656; G1 Survivor Space: 1593835520 - INFO 08:03:31 G1 Young Generation GC in 748ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 4386367472 -> 5874747352; G1 Survivor Space: 1627389952 -> INFO 08:09:30 Redistributing index summaries INFO 08:10:05 G1 Young Generation GC in 213ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 9276024272 -> 9969348048; G1 Survivor Space: 1711276032 - INFO 08:18:59 G1 Young Generation GC in 228ms. G1 Eden Space: 17347641344 -> 0; G1 Old Gen: 4546626040 -> 5180788216; G1 Survivor Space: 1593835520 - INFO 08:33:12 G1 Young Generation GC in 207ms. G1 Eden Space: 19411238912 -> 0; G1 Survivor Space: 1191182336 -> 1828716544; INFO 08:33:13 G1 Young Generation GC in 273ms. G1 Eden Space: 452984832 -> 0; G1 Old Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544 -> INFO 08:41:59 G1 Young Generation GC in 202ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8701765104 -> 8927144944; G1 Survivor Space: 1577058304 - INFO 08:56:31 G1 Young Generation GC in 205ms. G1 Eden Space: 11072962560 -> 0; G1 Old Gen: 9563013104 -> 11068430328; G1 Survivor Space: 1660944384 INFO 09:05:49 G1 Young Generation GC in 367ms. G1 Eden Space: 18522046464 -> 0; G1 Old Gen: 4009754608 -> 4732347872; G1 Survivor Space: 1778384896 - INFO 09:09:30 Redistributing index summaries INFO 09:14:52 G1 Young Generation GC in 213ms. G1 Eden Space: 15267266560 -> 0; G1 Old Gen: 12155092968 -> 12906235360; G1 Survivor Space: 1744830464 INFO 09:17:23 G1 Young Generation GC in 203ms. G1 Eden Space: 14159970304 -> 0; G1 Old Gen: 14779568608 -> 14771180008; G1 Survivor Space: 989855744 INFO 09:17:45 G1 Young Generation GC in 204ms. G1 Eden Space: 12616466432 -> 0; G1 Old Gen: 14771180008 -> 15500684768; G1 Survivor Space: 1761607680 INFO 09:20:52 G1 Young Generation GC in 361ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 4351990768 -> 5086824424; G1 Survivor Space: 1577058304 - INFO 09:23:51 G1 Young Generation GC in 207ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 7001879024 -> 7716560880; G1 Survivor Space: 1711276032 - INFO 09:29:20 G1 Young Generation GC in 202ms. G1 Eden Space: 17028874240 -> 0; G1 Survivor Space: 1040187392 -> 1828716544; INFO 09:29:40 G1 Young Generation GC in 219ms. G1 Eden Space: 15451815936 -> 0; G1 Old Gen: 11802771944 -> 12574988776; G1 Survivor Space: 1828716544 INFO 09:44:51 G1 Young Generation GC in 202ms. G1 Eden Space: 18001952768 -> 0; G1 Old Gen: 9325481456 -> 10085596664; G1 Survivor Space: 1795162112 INFO 09:45:23 G1 Young Generation GC in 209ms. G1 Eden Space: 9177137152 -> 0; G1 Old Gen: 10093985264 -> 12309118960; G1 Survivor Space: 1677721600 INFO 09:50:33 G1 Young Generation GC in 201ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 14487498224 -> 15268560880; G1 Survivor Space: 1761607680 INFO 10:08:09 Saved KeyCache (2989 items) in 92 ms INFO 10:09:30 Redistributing index summaries INFO 10:22:17 G1 Young Generation GC in 216ms. G1 Eden Space: 19495124992 -> 0; G1 Old Gen: 7742571504 -> 7734182904; G1 Survivor Space: 1107296256 - INFO 10:25:31 G1 Young Generation GC in 206ms. G1 Eden Space: 18622709760 -> 0; G1 Survivor Space: 1040187392 -> 1711276032; INFO 10:34:54 G1 Young Generation GC in 251ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 5066719736 -> 5691775992; G1 Survivor Space: 1610612736 - INFO 10:40:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 9000976880 -> 9090653168; G1 Survivor Space: 1442840576 - INFO 10:40:08 G1 Young Generation GC in 223ms. G1 Eden Space: 469762048 -> 0; G1 Old Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976 -> INFO 10:52:15 G1 Young Generation GC in 219ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 5331537912 -> 5828939768; G1 Survivor Space: 1275068416 - INFO 10:55:30 G1 Young Generation GC in 207ms. G1 Eden Space: 15804137472 -> 0; G1 Old Gen: 7501364216 -> 8001282032; G1 Survivor Space: 1459617792 - INFO 11:07:57 G1 Young Generation GC in 295ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4301379064 -> 4919709168; G1 Survivor Space: 1442840576 - INFO 11:09:30 Redistributing index summaries INFO 11:13:06 G1 Young Generation GC in 204ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 8044675064 -> 8659737080; G1 Survivor Space: 1476395008 - INFO 11:27:07 G1 Young Generation GC in 202ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4328521712 -> 4905718776; G1 Survivor Space: 1442840576 - INFO 11:30:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Survivor Space: 1358954496 -> 1828716544; INFO 11:30:25 G1 Young Generation GC in 226ms. G1 Eden Space: 14277410816 -> 0; G1 Old Gen: 6349296624 -> 7552639472; G1 Survivor Space: 1828716544 - INFO 11:32:30 G1 Young Generation GC in 215ms. G1 Eden Space: 19142803456 -> 0; G1 Survivor Space: 1459617792 -> 2046820352; INFO 11:33:01 G1 Young Generation GC in 247ms. G1 Eden Space: 17263755264 -> 0; G1 Old Gen: 7544250872 -> 8698295280; G1 Survivor Space: 2046820352 - INFO 11:46:29 G1 Young Generation GC in 355ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 4769358328 -> 5362416624; G1 Survivor Space: 1761607680 - INFO 11:49:18 G1 Young Generation GC in 233ms. G1 Eden Space: 17867735040 -> 0; G1 Old Gen: 6752829432 -> 6944805360; G1 Survivor Space: 1509949440 - INFO 11:52:24 G1 Young Generation GC in 209ms. G1 Eden Space: 19193135104 -> 0; G1 Survivor Space: 1409286144 -> 1946157056; INFO 11:52:47 G1 Young Generation GC in 338ms. G1 Eden Space: 18287165440 -> 0; G1 Old Gen: 8740881904 -> 9340426216; G1 Survivor Space: 1946157056 - INFO 12:04:08 G1 Young Generation GC in 810ms. G1 Eden Space: 18119393280 -> 0; G1 Old Gen: 3471267832 -> 5003518440; G1 Survivor Space: 1325400064 - INFO 12:06:28 G1 Young Generation GC in 345ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 5318377952 -> 6577762784; G1 Survivor Space: 1409286144 -> INFO 12:09:31 G1 Young Generation GC in 204ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 6577762784 -> 6586151384; G1 Survivor Space: 1442840576 - INFO 12:09:31 Redistributing index summaries INFO 12:10:01 G1 Young Generation GC in 298ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 6586151384 -> 7710758368; G1 Survivor Space: 1946157056 - INFO 12:12:37 G1 Young Generation GC in 214ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 8380219864 -> 8919123416; G1 Survivor Space: 1728053248 - INFO 12:24:54 G1 Young Generation GC in 299ms. G1 Eden Space: 16055795712 -> 0; G1 Old Gen: 4375908856 -> 5009812976; G1 Survivor Space: 1392508928 - INFO 12:28:34 G1 Young Generation GC in 485ms. G1 Eden Space: 13304332288 -> 0; G1 Old Gen: 6481607672 -> 7506585072; G1 Survivor Space: 1979711488 - INFO 12:44:52 G1 Young Generation GC in 261ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 4239590896 -> 4692442608; G1 Survivor Space: 1828716544 - INFO 12:48:37 G1 Young Generation GC in 251ms. G1 Eden Space: 13656653824 -> 0; G1 Old Gen: 6719275504 -> 7776254960; G1 Survivor Space: 1862270976 - INFO 12:53:47 G1 Young Generation GC in 217ms. G1 Eden Space: 17548967936 -> 0; G1 Old Gen: 9667142640 -> 10202159608; G1 Survivor Space: 1845493760 INFO 13:05:40 G1 Young Generation GC in 287ms. G1 Eden Space: 18320719872 -> 0; G1 Old Gen: 5298861040 -> 5826686456; G1 Survivor Space: 1862270976 - INFO 13:09:31 Redistributing index summaries INFO 13:11:44 G1 Young Generation GC in 268ms. G1 Eden Space: 10871635968 -> 0; G1 Old Gen: 8799649784 -> 9797878264; G1 Survivor Space: 1811939328 - INFO 13:14:22 G1 Young Generation GC in 202ms. G1 Eden Space: 17750294528 -> 0; G1 Survivor Space: 1325400064 -> 1795162112; INFO 13:14:40 G1 Young Generation GC in 254ms. G1 Eden Space: 14092861440 -> 0; G1 Old Gen: 10509551608 -> 11563311096; G1 Survivor Space: 1795162112 INFO 13:16:33 G1 Young Generation GC in 213ms. G1 Eden Space: 16039018496 -> 0; G1 Old Gen: 11619767792 -> 12115354104; G1 Survivor Space: 1627389952 INFO 13:25:40 G1 Young Generation GC in 245ms. G1 Eden Space: 18790481920 -> 0; G1 Old Gen: 4253024752 -> 4761733616; G1 Survivor Space: 1811939328 - INFO 13:27:38 G1 Young Generation GC in 208ms. G1 Eden Space: 13958643712 -> 0; G1 Old Gen: 5377097720 -> 6427987960; G1 Survivor Space: 1912602624 - INFO 13:29:30 G1 Young Generation GC in 207ms. G1 Eden Space: 12264144896 -> 0; G1 Old Gen: 6427987960 -> 7337681912; G1 Survivor Space: 1795162112 - INFO 13:33:32 G1 Young Generation GC in 202ms. G1 Eden Space: 18589155328 -> 0; G1 Old Gen: 8506316280 -> 8514704880; G1 Survivor Space: 1308622848 - INFO 13:35:09 G1 Young Generation GC in 203ms. G1 Eden Space: 17532190720 -> 0; G1 Survivor Space: 1258291200 -> 1761607680; INFO 13:38:33 G1 Young Generation GC in 216ms. G1 Eden Space: 11106516992 -> 0; G1 Old Gen: 11571635704 -> 12698973168; G1 Survivor Space: 1795162112 INFO 13:47:36 G1 Young Generation GC in 229ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3825205752 -> 4303660528; G1 Survivor Space: 1526726656 - INFO 13:53:43 G1 Young Generation GC in 207ms. G1 Eden Space: 14042529792 -> 0; G1 Old Gen: 7324985848 -> 8363582960; G1 Survivor Space: 1660944384 - INFO 13:55:40 G1 Young Generation GC in 201ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 8355194360 -> 8820888560; G1 Survivor Space: 1728053248 - INFO 13:56:22 G1 Young Generation GC in 206ms. G1 Eden Space: 19394461696 -> 0; G1 Survivor Space: 1207959552 -> 1711276032; INFO 14:08:09 Saved KeyCache (4563 items) in 189 ms INFO 14:09:31 Redistributing index summaries INFO 14:12:47 G1 Young Generation GC in 271ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4345299448 -> 4814148600; G1 Survivor Space: 1728053248 - INFO 14:18:05 G1 Young Generation GC in 224ms. G1 Eden Space: 18907922432 -> 0; G1 Old Gen: 7021265392 -> 7483297784; G1 Survivor Space: 1627389952 - INFO 14:36:05 G1 Young Generation GC in 304ms. G1 Eden Space: 18958254080 -> 0; G1 Old Gen: 3976200696 -> 4437455344; G1 Survivor Space: 1644167168 - INFO 14:36:07 G1 Young Generation GC in 212ms. G1 Eden Space: 1711276032 -> 0; G1 Old Gen: 4437455344 -> 5830082552; G1 Survivor Space: 1459617792 -> INFO 15:00:30 G1 Young Generation GC in 233ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4040399864 -> 4343952368; G1 Survivor Space: 1442840576 - INFO 15:09:31 Redistributing index summaries INFO 15:26:28 G1 Young Generation GC in 234ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4604513264 -> 5064485872; G1 Survivor Space: 1526726656 - INFO 15:52:08 G1 Young Generation GC in 222ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 4118806520 -> 4391313912; G1 Survivor Space: 1476395008 - INFO 15:55:06 G1 Young Generation GC in 202ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5267292152 -> 5559551472; INFO 16:09:31 Redistributing index summaries INFO 16:20:48 G1 Young Generation GC in 241ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 3854587896 -> 4287600112; G1 Survivor Space: 1476395008 - INFO 16:31:07 G1 Young Generation GC in 403ms. G1 Eden Space: 822083584 -> 0; G1 Old Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 -> 2 INFO 16:33:41 G1 Young Generation GC in 426ms. G1 Eden Space: 620756992 -> 0; G1 Old Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040 - INFO 16:38:17 G1 Young Generation GC in 835ms. G1 Eden Space: 251658240 -> 0; G1 Old Gen: 12062818288 -> 13404581312; G1 Survivor Space: 1459617792 - INFO 16:43:08 G1 Young Generation GC in 569ms. G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 14361297352 -> 15461740984; G1 Survivor Space: 1258291200 INFO 16:46:08 G1 Young Generation GC in 432ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 3256274416 -> 4355653104; G1 Survivor Space: 1023410176 - INFO 17:09:31 Redistributing index summaries INFO 17:26:10 G1 Young Generation GC in 744ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304 -> INFO 17:29:04 G1 Young Generation GC in 204ms. G1 Eden Space: 603979776 -> 0; G1 Old Gen: 8454412776 -> 9439609320; G1 Survivor Space: 1107296256 -> INFO 17:41:09 G1 Young Generation GC in 279ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 13615267816 -> 14860613592; G1 Survivor Space: 1358954496 - INFO 17:46:51 G1 Young Generation GC in 493ms. G1 Eden Space: 754974720 -> 0; G1 Old Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 -> 2 INFO 17:49:09 G1 Young Generation GC in 290ms. G1 Eden Space: 335544320 -> 0; G1 Old Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712 -> INFO 17:55:17 G1 Young Generation GC in 803ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304 -> INFO 17:58:07 G1 Young Generation GC in 206ms. G1 Eden Space: 4764729344 -> 0; G1 Old Gen: 8036691400 -> 8934491600; G1 Survivor Space: 973078528 -> INFO 18:07:29 G1 Young Generation GC in 306ms. G1 Eden Space: 704643072 -> 0; G1 Old Gen: 12163481536 -> 13045510592; G1 Survivor Space: 1006632960 - INFO 18:08:09 Saved KeyCache (5307 items) in 85 ms INFO 18:09:31 Redistributing index summaries INFO 18:10:11 G1 Young Generation GC in 277ms. G1 Eden Space: 402653184 -> 0; G1 Old Gen: 13045510592 -> 14252493256; G1 Survivor Space: 1308622848 - INFO 18:15:39 G1 Young Generation GC in 238ms. G1 Eden Space: 13052674048 -> 0; G1 Old Gen: 15141437888 -> 15520873416; G1 Survivor Space: 1358954496 INFO 18:20:16 G1 Young Generation GC in 540ms. G1 Eden Space: 7616856064 -> 0; G1 Old Gen: 3909694456 -> 4886048752; G1 Survivor Space: 1509949440 -> INFO 18:23:04 G1 Young Generation GC in 221ms. G1 Eden Space: 16894656512 -> 0; G1 Old Gen: 5420089848 -> 6187322368; G1 Survivor Space: 939524096 -> INFO 18:42:08 G1 Young Generation GC in 708ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496 - INFO 18:50:13 G1 Young Generation GC in 318ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224 -> INFO 19:09:31 Redistributing index summaries INFO 19:27:02 G1 Young Generation GC in 384ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 7034014192 -> 7759489528; G1 Survivor Space: 889192448 -> INFO 19:31:25 G1 Young Generation GC in 770ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 8506048488 -> 9921951720; G1 Survivor Space: 1493172224 -> INFO 20:09:31 Redistributing index summaries INFO 21:09:31 Redistributing index summaries INFO 21:19:39 G1 Young Generation GC in 201ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3917480432 -> 4124623344; G1 Survivor Space: 1459617792 - INFO 22:04:05 G1 Young Generation GC in 209ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 4420796912 -> 4618141680; G1 Survivor Space: 1459617792 - INFO 22:07:49 G1 Young Generation GC in 215ms. G1 Eden Space: 14713618432 -> 0; G1 Old Gen: 5970255352 -> 6555802608; G1 Survivor Space: 1493172224 - INFO 22:08:09 Saved KeyCache (5845 items) in 87 ms INFO 22:09:31 Redistributing index summaries INFO 22:38:04 G1 Young Generation GC in 201ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4248291312 -> 4459621368; G1 Survivor Space: 1526726656 - INFO 22:44:12 G1 Young Generation GC in 203ms. G1 Eden Space: 10770972672 -> 0; G1 Old Gen: 6703617016 -> 7549747184; G1 Survivor Space: 1325400064 - INFO 22:47:05 G1 Young Generation GC in 207ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 7549747184 -> 7771343352; G1 Survivor Space: 1577058304 - INFO 23:08:59 G1 Young Generation GC in 369ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 4823449592 -> 5478502888; G1 Survivor Space: 1660944384 - INFO 23:09:31 Redistributing index summaries INFO 23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 23:58:40 G1 Young Generation GC in 283ms. G1 Eden Space: 18941476864 -> 0; G1 Old Gen: 4831838192 -> 5297926640; G1 Survivor Space: 1660944384 - INFO 00:01:54 G1 Young Generation GC in 211ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 6425673712 -> 6912948208; G1 Survivor Space: 1711276032 - INFO 00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:09:31 Redistributing index summaries INFO 00:14:33 G1 Young Generation GC in 282ms. G1 Eden Space: 14344519680 -> 0; G1 Old Gen: 14159246832 -> 14677156848; G1 Survivor Space: 1207959552 INFO 00:21:32 G1 Young Generation GC in 225ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 4346926576 -> 4865982456; G1 Survivor Space: 1560281088 - INFO 00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:24:31 G1 Young Generation GC in 266ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6106907128 -> 6613907952; G1 Survivor Space: 1593835520 - INFO 00:29:46 G1 Young Generation GC in 233ms. G1 Eden Space: 18236833792 -> 0; G1 Old Gen: 9076474360 -> 9498544624; G1 Survivor Space: 1795162112 - INFO 00:32:35 G1 Young Generation GC in 226ms. G1 Eden Space: 16492003328 -> 0; G1 Old Gen: 10804527088 -> 11351712752; G1 Survivor Space: 1811939328 INFO 00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:41:24 G1 Young Generation GC in 330ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3426117112 -> 3961507312; G1 Survivor Space: 1459617792 - INFO 00:44:26 G1 Young Generation GC in 325ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 5268046320 -> 5788210120; G1 Survivor Space: 1560281088 - INFO 00:47:13 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 7147093960 -> 7138705360; G1 Survivor Space: 1358954496 - INFO 00:47:28 G1 Young Generation GC in 312ms. G1 Eden Space: 18605932544 -> 0; G1 Old Gen: 7138705360 -> 7710094280; G1 Survivor Space: 1996488704 - INFO 00:50:35 G1 Young Generation GC in 330ms. G1 Eden Space: 17934843904 -> 0; G1 Old Gen: 9118416840 -> 9688820680; G1 Survivor Space: 1929379840 - INFO 00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 00:56:29 G1 Young Generation GC in 251ms. G1 Eden Space: 14797504512 -> 0; G1 Old Gen: 13404995520 -> 13787049928; G1 Survivor Space: 1358954496 INFO 00:58:14 G1 Young Generation GC in 662ms. G1 Eden Space: 369098752 -> 0; G1 Old Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280 - INFO 01:01:25 G1 Young Generation GC in 288ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 2618580968 -> 3127466464; G1 Survivor Space: 1560281088 - INFO 01:04:17 G1 Young Generation GC in 201ms. G1 Eden Space: 19327352832 -> 0; G1 Old Gen: 4521459688 -> 4529848288; G1 Survivor Space: 1275068416 - INFO 01:04:39 G1 Young Generation GC in 283ms. G1 Eden Space: 18723373056 -> 0; G1 Old Gen: 4529848288 -> 5024210408; G1 Survivor Space: 1879048192 - INFO 01:07:23 G1 Young Generation GC in 294ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 6308233704 -> 6875745256; G1 Survivor Space: 1862270976 - INFO 01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 01:09:31 Redistributing index summaries INFO 01:10:12 G1 Young Generation GC in 213ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 8262779360 -> 8254390760; G1 Survivor Space: 1358954496 - INFO 01:10:33 G1 Young Generation GC in 319ms. G1 Eden Space: 18639486976 -> 0; G1 Old Gen: 8254390760 -> 8821123552; G1 Survivor Space: 1962934272 - INFO 01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 01:24:25 G1 Young Generation GC in 227ms. G1 Eden Space: 19226689536 -> 0; G1 Old Gen: 4245307376 -> 4236918776; G1 Survivor Space: 1375731712 - INFO 01:24:48 G1 Young Generation GC in 219ms. G1 Eden Space: 15401484288 -> 0; G1 Old Gen: 4236918776 -> 5231503864; G1 Survivor Space: 2097152000 - INFO 01:42:01 G1 Young Generation GC in 212ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4320133112 -> 5100782576; G1 Survivor Space: 1627389952 - INFO 01:48:28 G1 Young Generation GC in 209ms. G1 Eden Space: 14730395648 -> 0; G1 Old Gen: 8901994480 -> 9402393592; G1 Survivor Space: 1291845632 - INFO 02:04:18 G1 Young Generation GC in 241ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7825270264 -> 8327173624; G1 Survivor Space: 1560281088 - INFO 02:08:09 Saved KeyCache (8634 items) in 94 ms INFO 02:09:31 Redistributing index summaries INFO 02:14:50 G1 Young Generation GC in 314ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3505807352 -> 4140375528; G1 Survivor Space: 1526726656 - INFO 02:48:35 G1 Young Generation GC in 321ms. G1 Eden Space: 11576279040 -> 0; G1 Old Gen: 3865890800 -> 4482588664; G1 Survivor Space: 1543503872 - INFO 02:53:54 G1 Young Generation GC in 203ms. G1 Eden Space: 17901289472 -> 0; G1 Old Gen: 7927235056 -> 8552357368; G1 Survivor Space: 1593835520 - INFO 03:04:33 G1 Young Generation GC in 345ms. G1 Eden Space: 13505658880 -> 0; G1 Old Gen: 3765058032 -> 4443454952; G1 Survivor Space: 1442840576 - INFO 03:09:31 Redistributing index summaries INFO 03:22:56 G1 Young Generation GC in 250ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 4806672880 -> 5340134904; G1 Survivor Space: 1610612736 - INFO 03:35:13 G1 Young Generation GC in 210ms. G1 Eden Space: 14445182976 -> 0; G1 Survivor Space: 1073741824 -> 1744830464; INFO 03:38:42 G1 Young Generation GC in 214ms. G1 Eden Space: 18303942656 -> 0; G1 Old Gen: 4786540536 -> 5432116720; G1 Survivor Space: 1375731712 - INFO 03:52:11 G1 Young Generation GC in 207ms. G1 Eden Space: 117440512 -> 0; G1 Old Gen: 12054831088 -> 10154180600; G1 Survivor Space: 1593835520 - INFO 03:54:48 G1 Young Generation GC in 224ms. G1 Eden Space: 16156459008 -> 0; G1 Old Gen: 4303355896 -> 4801558520; G1 Survivor Space: 1560281088 - INFO 04:09:31 Redistributing index summaries INFO 04:16:09 G1 Young Generation GC in 216ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5853311984 -> 6366188536; G1 Survivor Space: 1459617792 - INFO 04:19:13 G1 Young Generation GC in 229ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7826571256 -> 8457920504; G1 Survivor Space: 1560281088 - INFO 04:27:41 G1 Young Generation GC in 311ms. G1 Eden Space: 13723762688 -> 0; G1 Old Gen: 14176747504 -> 14808230392; G1 Survivor Space: 1509949440 INFO 04:29:56 G1 Young Generation GC in 274ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 3376452080 -> 3818638328; G1 Survivor Space: 1459617792 - INFO 04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB INFO 04:49:07 G1 Young Generation GC in 319ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4435714040 -> 5023449072; G1 Survivor Space: 1442840576 - INFO 04:54:06 G1 Young Generation GC in 208ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 7843348472 -> 8466889208; G1 Survivor Space: 1476395008 - INFO 05:09:31 Redistributing index summaries I found INFO Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB, does it mean the memory is not enough ? I want to know why adding a new node spent so much time and what need I do to complete a new node joining the cluster ? I need some help and really need some suggestions. Thanks a lot!
