Hello, You don't have to use synchronization for Ignite Cache API. This instance already is threadsafe. I recommend to read the following documentation page: https://apacheignite.readme.io/docs/jvm-and-system-tuning
On Sun, Dec 10, 2017 at 4:49 PM, Aurora <[email protected]> wrote: > @Nikolai > Your suggestion is right. > "to-space exhausted" occured in GC Logs on 3 out of 6 server nodes, as: > Node 1: > 2017-12-05T18:36:48.716+0800: 6870.705: [GC pause (G1 Evacuation Pause) > (young), 0.4171797 secs] > [Parallel Time: 412.1 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 6870705.5, Avg: 6870705.7, Max: > 6870705.8, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.4, > Sum: > 9.6] > [Update RS (ms): Min: 56.8, Avg: 58.6, Max: 60.2, Diff: 3.4, Sum: > 761.8] > [Processed Buffers: Min: 69, Avg: 76.7, Max: 91, Diff: 22, Sum: > 997] > [Scan RS (ms): Min: 1.4, Avg: 2.8, Max: 4.7, Diff: 3.3, Sum: 36.7] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, > Sum: 0.2] > [Object Copy (ms): Min: 349.1, Avg: 349.2, Max: 349.3, Diff: 0.2, > Sum: > 4540.2] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5] > [Termination Attempts: Min: 1, Avg: 40.3, Max: 75, Diff: 74, Sum: > 524] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: > 1.7] > [GC Worker Total (ms): Min: 411.4, Avg: 411.6, Max: 411.8, Diff: 0.4, > Sum: 5350.7] > [GC Worker End (ms): Min: 6871117.2, Avg: 6871117.3, Max: 6871117.4, > Diff: 0.2] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.9 ms] > [Other: 4.1 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.4 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 0.9 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 1.9 ms] > [Eden: 8280.0M(8280.0M)->0.0B(8032.0M) Survivors: 968.0M->968.0M Heap: > 13.4G(16.0G)->5680.8M(16.0G)] > [Times: user=5.37 sys=0.00, real=0.42 secs] > 2017-12-05T18:40:45.218+0800: 7107.207: [GC pause (G1 Evacuation Pause) > (young) (to-space exhausted), 19.5379052 secs] > [Parallel Time: 16666.0 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 7107207.2, Avg: 7107207.4, Max: > 7107207.5, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, > Sum: > 9.1] > [Update RS (ms): Min: 143.9, Avg: 144.0, Max: 144.1, Diff: 0.2, Sum: > 1871.6] > [Processed Buffers: Min: 78, Avg: 87.2, Max: 108, Diff: 30, Sum: > 1134] > [Scan RS (ms): Min: 6.1, Avg: 6.2, Max: 6.3, Diff: 0.2, Sum: 80.3] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.2] > [Object Copy (ms): Min: 16474.9, Avg: 16481.7, Max: 16514.7, Diff: > 39.8, Sum: 214262.3] > [Termination (ms): Min: 0.0, Avg: 32.9, Max: 39.6, Diff: 39.6, Sum: > 427.8] > [Termination Attempts: Min: 1, Avg: 1.5, Max: 4, Diff: 3, Sum: 19] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: > 0.9] > [GC Worker Total (ms): Min: 16665.4, Avg: 16665.6, Max: 16665.7, > Diff: > 0.3, Sum: 216652.3] > [GC Worker End (ms): Min: 7123872.9, Avg: 7123872.9, Max: 7123873.0, > Diff: 0.1] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.8 ms] > [Other: 2871.0 ms] > [Evacuation Failure: 2861.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.4 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 6.9 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 1.3 ms] > [Eden: 8032.0M(8032.0M)->0.0B(328.0M) Survivors: 968.0M->1504.0M Heap: > 13.5G(16.0G)->12.5G(16.0G)] > [Times: user=43.38 sys=7.36, real=19.53 secs] > 2017-12-05T18:41:05.194+0800: 7127.183: [GC pause (G1 Evacuation Pause) > (young) (initial-mark), 0.9987695 secs] > [Parallel Time: 990.4 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 7127183.8, Avg: 7127183.9, Max: > 7127184.0, > Diff: 0.2] > [Ext Root Scanning (ms): Min: 0.9, Avg: 1.1, Max: 1.3, Diff: 0.4, > Sum: > 14.3] > [Update RS (ms): Min: 563.4, Avg: 563.6, Max: 564.4, Diff: 1.0, Sum: > 7327.2] > [Processed Buffers: Min: 1025, Avg: 1339.8, Max: 1575, Diff: 550, > Sum: 17418] > [Scan RS (ms): Min: 15.0, Avg: 15.7, Max: 16.0, Diff: 1.0, Sum: > 204.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.1] > [Object Copy (ms): Min: 409.2, Avg: 409.4, Max: 409.4, Diff: 0.2, > Sum: > 5321.6] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] > [Termination Attempts: Min: 1, Avg: 33.3, Max: 48, Diff: 47, Sum: > 433] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: > 1.2] > [GC Worker Total (ms): Min: 989.8, Avg: 990.0, Max: 990.2, Diff: 0.4, > Sum: 12869.6] > [GC Worker End (ms): Min: 7128173.8, Avg: 7128173.9, Max: 7128174.0, > Diff: 0.2] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.3 ms] > [Other: 7.9 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.4 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 4.8 ms] > [Humongous Register: 0.0 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.8 ms] > [Eden: 328.0M(328.0M)->0.0B(336.0M) Survivors: 1504.0M->280.0M Heap: > 12.9G(16.0G)->12.8G(16.0G)] > [Times: user=12.93 sys=0.00, real=1.00 secs] > > Node 2: > 2017-12-05T18:37:14.177+0800: 9497.100: [GC pause (G1 Evacuation Pause) > (young), 0.2761496 secs] > [Parallel Time: 271.9 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 9497100.0, Avg: 9497100.2, Max: > 9497100.4, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.6, Avg: 0.8, Max: 0.9, Diff: 0.4, > Sum: > 10.0] > [Update RS (ms): Min: 19.0, Avg: 20.3, Max: 22.0, Diff: 3.0, Sum: > 264.3] > [Processed Buffers: Min: 6, Avg: 13.7, Max: 42, Diff: 36, Sum: > 178] > [Scan RS (ms): Min: 0.1, Avg: 0.3, Max: 0.7, Diff: 0.7, Sum: 3.5] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, > Sum: 0.2] > [Object Copy (ms): Min: 247.7, Avg: 249.3, Max: 250.5, Diff: 2.7, > Sum: > 3241.1] > [Termination (ms): Min: 0.0, Avg: 0.5, Max: 0.7, Diff: 0.7, Sum: 6.9] > [Termination Attempts: Min: 1, Avg: 374.3, Max: 476, Diff: 475, > Sum: 4866] > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: > 2.1] > [GC Worker Total (ms): Min: 271.1, Avg: 271.4, Max: 271.7, Diff: 0.6, > Sum: 3528.1] > [GC Worker End (ms): Min: 9497371.5, Avg: 9497371.6, Max: 9497371.7, > Diff: 0.3] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.9 ms] > [Other: 3.2 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.3 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 0.4 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 2.0 ms] > [Eden: 9392.0M(9392.0M)->0.0B(9080.0M) Survivors: 432.0M->744.0M Heap: > 13.5G(16.0G)->4752.0M(16.0G)] > [Times: user=3.54 sys=0.01, real=0.28 secs] > 2017-12-05T18:43:29.983+0800: 9872.905: [GC pause (G1 Evacuation Pause) > (young) (to-space exhausted), 45.1101748 secs] > [Parallel Time: 39495.6 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 9872905.5, Avg: 9872905.7, Max: > 9872905.8, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.6, Avg: 0.8, Max: 0.9, Diff: 0.3, > Sum: > 9.8] > [Update RS (ms): Min: 1180.8, Avg: 1183.2, Max: 1184.5, Diff: 3.7, > Sum: 15382.2] > [Processed Buffers: Min: 63, Avg: 73.8, Max: 95, Diff: 32, Sum: > 960] > [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 2.8] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.2] > [Object Copy (ms): Min: 38287.7, Avg: 38290.6, Max: 38310.0, Diff: > 22.3, Sum: 497777.7] > [Termination (ms): Min: 0.0, Avg: 20.3, Max: 22.0, Diff: 22.0, Sum: > 263.9] > [Termination Attempts: Min: 1, Avg: 20.8, Max: 32, Diff: 31, Sum: > 271] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: > 1.3] > [GC Worker Total (ms): Min: 39495.0, Avg: 39495.2, Max: 39495.5, > Diff: > 0.4, Sum: 513437.7] > [GC Worker End (ms): Min: 9912400.8, Avg: 9912400.9, Max: 9912401.0, > Diff: 0.2] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.8 ms] > [Other: 5613.6 ms] > [Evacuation Failure: 5603.4 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.4 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 8.1 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 1.0 ms] > [Eden: 9080.0M(9080.0M)->0.0B(8192.0K) Survivors: 744.0M->1640.0M Heap: > 13.6G(16.0G)->13.2G(16.0G)] > [Times: user=75.59 sys=19.17, real=45.10 secs] > 2017-12-05T18:44:15.116+0800: 9918.039: [GC pause (G1 Evacuation Pause) > (young) (initial-mark), 1.2374672 secs] > [Parallel Time: 1229.6 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 9918039.4, Avg: 9918039.5, Max: > 9918039.6, > Diff: 0.2] > [Ext Root Scanning (ms): Min: 0.8, Avg: 0.9, Max: 1.2, Diff: 0.4, > Sum: > 12.3] > [Update RS (ms): Min: 855.4, Avg: 855.5, Max: 855.7, Diff: 0.4, Sum: > 11121.4] > [Processed Buffers: Min: 1844, Avg: 2442.1, Max: 2840, Diff: 996, > Sum: 31747] > [Scan RS (ms): Min: 13.3, Avg: 13.4, Max: 13.5, Diff: 0.3, Sum: > 174.2] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.1] > [Object Copy (ms): Min: 359.2, Avg: 359.2, Max: 359.3, Diff: 0.1, > Sum: > 4669.9] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] > [Termination Attempts: Min: 1, Avg: 19.1, Max: 33, Diff: 32, Sum: > 248] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: > 1.4] > [GC Worker Total (ms): Min: 1229.0, Avg: 1229.2, Max: 1229.4, Diff: > 0.4, Sum: 15979.5] > [GC Worker End (ms): Min: 9919268.6, Avg: 9919268.7, Max: 9919268.8, > Diff: 0.2] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.3 ms] > [Other: 7.4 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.4 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 4.7 ms] > [Humongous Register: 0.0 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.8 ms] > [Eden: 8192.0K(8192.0K)->0.0B(456.0M) Survivors: 1640.0M->56.0M Heap: > 13.2G(16.0G)->13.3G(16.0G)] > [Times: user=16.03 sys=0.00, real=1.24 secs] > 2017-12-05T18:44:16.354+0800: 9919.276: [GC > concurrent-root-region-scan-start] > 2017-12-05T18:44:16.368+0800: 9919.291: [GC concurrent-root-region-scan- > end, > 0.0141460 secs] > 2017-12-05T18:44:16.368+0800: 9919.291: [GC concurrent-mark-start] > 2017-12-05T18:44:19.218+0800: 9922.141: [GC pause (G1 Evacuation Pause) > (young), 0.1909109 secs] > [Parallel Time: 187.1 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 9922141.8, Avg: 9922141.9, Max: > 9922142.1, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, > Sum: > 9.6] > [Update RS (ms): Min: 54.0, Avg: 54.2, Max: 54.5, Diff: 0.5, Sum: > 704.3] > [Processed Buffers: Min: 58, Avg: 64.4, Max: 80, Diff: 22, Sum: > 837] > [Scan RS (ms): Min: 4.9, Avg: 5.2, Max: 5.4, Diff: 0.5, Sum: 68.1] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 126.3, Avg: 126.3, Max: 126.4, Diff: 0.2, > Sum: > 1641.9] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.6] > [Termination Attempts: Min: 1, Avg: 77.5, Max: 97, Diff: 96, Sum: > 1007] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: > 1.0] > [GC Worker Total (ms): Min: 186.5, Avg: 186.7, Max: 186.9, Diff: 0.4, > Sum: 2426.6] > [GC Worker End (ms): Min: 9922328.6, Avg: 9922328.6, Max: 9922328.7, > Diff: 0.1] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.3 ms] > [Other: 3.4 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.5 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 1.2 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.4 ms] > [Eden: 456.0M(456.0M)->0.0B(424.0M) Survivors: 56.0M->88.0M Heap: > 13.8G(16.0G)->13.7G(16.0G)] > [Times: user=2.43 sys=0.01, real=0.19 secs] > > Node 3: > 2017-12-05T18:41:07.782+0800: 7118.035: [GC pause (G1 Evacuation Pause) > (young), 0.5114032 secs] > [Parallel Time: 506.9 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 7118034.9, Avg: 7118035.0, Max: > 7118035.2, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, > Sum: > 9.3] > [Update RS (ms): Min: 50.6, Avg: 51.5, Max: 52.5, Diff: 1.9, Sum: > 669.4] > [Processed Buffers: Min: 20, Avg: 31.3, Max: 56, Diff: 36, Sum: > 407] > [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.6, Diff: 0.6, Sum: 3.0] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.2] > [Object Copy (ms): Min: 453.0, Avg: 453.8, Max: 454.3, Diff: 1.3, > Sum: > 5899.5] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5] > [Termination Attempts: Min: 1, Avg: 20.4, Max: 27, Diff: 26, Sum: > 265] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: > 1.8] > [GC Worker Total (ms): Min: 506.2, Avg: 506.4, Max: 506.7, Diff: 0.5, > Sum: 6583.7] > [GC Worker End (ms): Min: 7118541.4, Avg: 7118541.5, Max: 7118541.6, > Diff: 0.2] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.9 ms] > [Other: 3.5 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.5 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 0.5 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 1.8 ms] > [Eden: 9208.0M(9208.0M)->0.0B(7160.0M) Survivors: 384.0M->1600.0M Heap: > 13.5G(16.0G)->5928.0M(16.0G)] > [Times: user=6.59 sys=0.00, real=0.52 secs] > 2017-12-05T18:59:32.939+0800: 8223.192: [GC pause (G1 Evacuation Pause) > (young) (to-space exhausted), 55.7872972 secs] > [Parallel Time: 50147.5 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 8223192.1, Avg: 8223192.2, Max: > 8223192.4, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.6, Avg: 0.7, Max: 0.9, Diff: 0.3, > Sum: > 9.2] > [Update RS (ms): Min: 514.3, Avg: 516.5, Max: 523.9, Diff: 9.6, Sum: > 6714.9] > [Processed Buffers: Min: 105, Avg: 122.5, Max: 165, Diff: 60, Sum: > 1593] > [Scan RS (ms): Min: 1916.0, Avg: 1923.1, Max: 1925.4, Diff: 9.5, Sum: > 25000.0] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.0] > [Object Copy (ms): Min: 47705.9, Avg: 47706.4, Max: 47707.0, Diff: > 1.0, Sum: 620183.4] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 1.8] > [Termination Attempts: Min: 1, Avg: 28.2, Max: 47, Diff: 46, Sum: > 367] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: > 1.0] > [GC Worker Total (ms): Min: 50146.9, Avg: 50147.0, Max: 50147.2, > Diff: > 0.3, Sum: 651911.3] > [GC Worker End (ms): Min: 8273339.2, Avg: 8273339.3, Max: 8273339.3, > Diff: 0.2] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.6 ms] > [Other: 5639.1 ms] > [Evacuation Failure: 5622.5 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.3 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 12.1 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 3.7 ms] > [Eden: 7160.0M(7160.0M)->0.0B(8192.0K) Survivors: 1600.0M->1232.0M > Heap: > 13.1G(16.0G)->13.8G(16.0G)] > [Times: user=96.95 sys=24.80, real=55.78 secs] > 2017-12-05T19:00:28.876+0800: 8279.128: [GC pause (G1 Evacuation Pause) > (young) (initial-mark), 2.3525928 secs] > [Parallel Time: 2345.5 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 8279129.3, Avg: 8279129.4, Max: > 8279129.5, > Diff: 0.2] > [Ext Root Scanning (ms): Min: 0.7, Avg: 0.8, Max: 1.0, Diff: 0.3, > Sum: > 10.9] > [Update RS (ms): Min: 1933.1, Avg: 1933.3, Max: 1933.6, Diff: 0.5, > Sum: 25132.7] > [Processed Buffers: Min: 2505, Avg: 3215.6, Max: 3919, Diff: 1414, > Sum: 41803] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.5] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, > Sum: 0.1] > [Object Copy (ms): Min: 410.4, Avg: 410.7, Max: 410.9, Diff: 0.5, > Sum: > 5338.6] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] > [Termination Attempts: Min: 1, Avg: 84.6, Max: 120, Diff: 119, > Sum: > 1100] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: > 1.2] > [GC Worker Total (ms): Min: 2344.9, Avg: 2345.1, Max: 2345.2, Diff: > 0.3, Sum: 30485.8] > [GC Worker End (ms): Min: 8281474.4, Avg: 8281474.5, Max: 8281474.6, > Diff: 0.2] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.4 ms] > [Other: 6.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.7 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 3.5 ms] > [Humongous Register: 0.0 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.4 ms] > [Eden: 8192.0K(8192.0K)->0.0B(472.0M) Survivors: 1232.0M->40.0M Heap: > 13.8G(16.0G)->13.9G(16.0G)] > [Times: user=30.54 sys=0.00, real=2.35 secs] > 2017-12-05T19:00:31.229+0800: 8281.481: [GC > concurrent-root-region-scan-start] > 2017-12-05T19:00:31.230+0800: 8281.482: [GC concurrent-root-region-scan- > end, > 0.0008827 secs] > 2017-12-05T19:00:31.230+0800: 8281.482: [GC concurrent-mark-start] > 2017-12-05T19:00:31.332+0800: 8281.585: [GC concurrent-mark-end, 0.1025943 > secs] > 2017-12-05T19:00:31.334+0800: 8281.586: [GC remark > 2017-12-05T19:00:31.334+0800: 8281.586: [Finalize Marking, 0.0005019 secs] > 2017-12-05T19:00:31.334+0800: 8281.587: [GC ref-proc, 0.0009665 secs] > 2017-12-05T19:00:31.335+0800: 8281.588: [Unloading, 0.0061329 secs], > 0.0095552 secs] > [Times: user=0.08 sys=0.00, real=0.01 secs] > 2017-12-05T19:00:31.344+0800: 8281.596: [GC cleanup 13G->4938M(16G), > 0.0225853 secs] > [Times: user=0.26 sys=0.00, real=0.02 secs] > 2017-12-05T19:00:31.367+0800: 8281.619: [GC concurrent-cleanup-start] > 2017-12-05T19:00:31.375+0800: 8281.628: [GC concurrent-cleanup-end, > 0.0086164 secs] > 2017-12-05T19:24:35.312+0800: 9725.565: [GC pause (G1 Evacuation Pause) > (young), 0.0602876 secs] > [Parallel Time: 57.3 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 9725566.0, Avg: 9725566.2, Max: > 9725566.4, > Diff: 0.3] > [Ext Root Scanning (ms): Min: 0.5, Avg: 0.7, Max: 0.9, Diff: 0.4, > Sum: > 8.6] > [Update RS (ms): Min: 44.0, Avg: 44.7, Max: 46.1, Diff: 2.1, Sum: > 580.6] > [Processed Buffers: Min: 39, Avg: 75.1, Max: 97, Diff: 58, Sum: > 976] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 9.9, Avg: 11.4, Max: 12.0, Diff: 2.1, Sum: > 148.3] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] > [Termination Attempts: Min: 1, Avg: 3.3, Max: 7, Diff: 6, Sum: 43] > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: > 0.6] > [GC Worker Total (ms): Min: 56.6, Avg: 56.8, Max: 57.0, Diff: 0.4, > Sum: 738.3] > [GC Worker End (ms): Min: 9725623.0, Avg: 9725623.0, Max: 9725623.0, > Diff: 0.1] > [Code Root Fixup: 0.1 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.3 ms] > [Other: 2.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.3 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 0.4 ms] > [Humongous Register: 0.1 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.2 ms] > [Eden: 472.0M(472.0M)->0.0B(456.0M) Survivors: 40.0M->56.0M Heap: > 5404.0M(16.0G)->4952.0M(16.0G)] > [Times: user=0.74 sys=0.00, real=0.06 secs] > > From the above logs, we could find that GC on 3 nodes haven't worked one by > one. why? > > My GC configuration in ignite.sh(JDK 8) is: > JVM_OPTS="-Xms16g -Xmx16g -server -XX:+AggressiveOpts -XX:MaxPermSize=256m" > > JVM_OPTS="$JVM_OPTS -XX:NewSize=512m -XX:SurvivorRatio=6 > -XX:+AlwaysPreTouch > -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 -XX:GCTimeRatio=4 > -XX:InitiatingHeapOccupancyPercent=30 -XX:G1HeapRegionSize=8M > -XX:ConcGCThreads=4 -XX:G1HeapWastePercent=10 -XX:+UseTLAB > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC > > Do I need to improve it following this > post:https://stackoverflow.com/questions/30796111/g1- > young-gc-does-not-free-memory-to-space-exhausted? > > Besides, I am not sure whether those codes consuming kafka messages are > correct or not, like: > public static Object person_object = new Object(); > synchronized(person_object){ > personCache.put(personKey, person); > } > > Does this lock affect those cache opertators on server nodes? > > Thanks. > > > > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
