Nice to see another success of G1 GC :) Jörg
On Thu, Sep 11, 2014 at 11:34 AM, Robert Gardam <[email protected]> wrote: > Yeah, we are seeing much better gc performance here too! > > We were experiencing the stop the world GC with CMS and then nodes would > time out. > > Our heap is 32gb, we run two nodes per system. IO doesn't seem to be an > issue here. > 20 nodes = 10 boxes, 128gb ram each > > Field cache is limited to 20% > We're bulk indexing around 10k events/s > > It seems to be much more stable and predictable in terms of GC. The GC > Logs are showing a huge reductions > > 61043,866: Total time for which application threads were stopped: > 0,0140156 seconds > 61044,524: Total time for which application threads were stopped: > 0,0005284 seconds > 61045,801: Total time for which application threads were stopped: > 0,0006138 seconds > 61045,802: Total time for which application threads were stopped: > 0,0003635 seconds > 61045,802: Total time for which application threads were stopped: > 0,0002545 seconds > 61045,803: Total time for which application threads were stopped: > 0,0002944 seconds > 61045,804: Total time for which application threads were stopped: > 0,0002367 seconds > 61046,469: Total time for which application threads were stopped: > 0,0004653 seconds > 61048,172: Total time for which application threads were stopped: > 0,0004850 seconds > 61048,598: Total time for which application threads were stopped: > 0,0004937 seconds > 61049,197: Total time for which application threads were stopped: > 0,0004396 seconds > 61050,264: Total time for which application threads were stopped: > 0,0004587 seconds > 61051,593: Total time for which application threads were stopped: > 0,0004600 seconds > 61051,689: Total time for which application threads were stopped: > 0,0005021 seconds > 61053,822: Total time for which application threads were stopped: > 0,0004721 seconds > 61053,824: Total time for which application threads were stopped: > 0,0005323 seconds > 61053,825: Total time for which application threads were stopped: > 0,0003403 seconds > 61053,825: Total time for which application threads were stopped: > 0,0003301 seconds > 61053,826: Total time for which application threads were stopped: > 0,0003322 seconds > 61053,826: Total time for which application threads were stopped: > 0,0003364 seconds > 61059,265: Total time for which application threads were stopped: > 0,0004321 seconds > 61061,691: Total time for which application threads were stopped: > 0,0004619 seconds > 61062,595: Total time for which application threads were stopped: > 0,0004529 seconds > 61064,199: Total time for which application threads were stopped: > 0,0004587 seconds > 61070,267: Total time for which application threads were stopped: > 0,0004606 seconds > 61074,200: Total time for which application threads were stopped: > 0,0004508 seconds > 61076,693: Total time for which application threads were stopped: > 0,0004709 seconds > 61077,597: Total time for which application threads were stopped: > 0,0004698 seconds > 61079,268: Total time for which application threads were stopped: > 0,0004601 seconds > 61079,817: Total time for which application threads were stopped: > 0,0004535 seconds > 61081,818: Total time for which application threads were stopped: > 0,0004979 seconds > 61082,819: Total time for which application threads were stopped: > 0,0004817 seconds > 61089,204: Total time for which application threads were stopped: > 0,0011584 seconds > 61091,699: Total time for which application threads were stopped: > 0,0004501 seconds > 61092,599: Total time for which application threads were stopped: > 0,0004539 seconds > 61094,204: Total time for which application threads were stopped: > 0,0006452 seconds > 61095,271: Total time for which application threads were stopped: > 0,0006568 seconds > 61101,701: Total time for which application threads were stopped: > 0,0004679 seconds > 61102,601: Total time for which application threads were stopped: > 0,0004576 seconds > 61104,272: Total time for which application threads were stopped: > 0,0004474 seconds > 61114,207: Total time for which application threads were stopped: > 0,0005483 seconds > 61115,273: Total time for which application threads were stopped: > 0,0004848 seconds > 61117,604: Total time for which application threads were stopped: > 0,0008780 seconds > 61117,703: Total time for which application threads were stopped: > 0,0005068 seconds > 61124,274: Total time for which application threads were stopped: > 0,0004519 seconds > 61127,605: Total time for which application threads were stopped: > 0,0004786 seconds > 61129,249: [GC pause (G1 Evacuation Pause) (young) > Desired survivor size 1291845632 bytes, new threshold 15 (max 15) > - age 1: 19633616 bytes, 19633616 total > - age 2: 3428232 bytes, 23061848 total > - age 3: 1362152 bytes, 24424000 total > - age 4: 1443728 bytes, 25867728 total > - age 5: 996840 bytes, 26864568 total > - age 6: 1584400 bytes, 28448968 total > - age 7: 1697168 bytes, 30146136 total > - age 8: 578056 bytes, 30724192 total > - age 9: 1166056 bytes, 31890248 total > - age 10: 8904 bytes, 31899152 total > - age 11: 31640 bytes, 31930792 total > - age 12: 979168 bytes, 32909960 total > - age 13: 108016 bytes, 33017976 total > - age 14: 15376 bytes, 33033352 total > - age 15: 755696 bytes, 33789048 total > , 0,0134055 secs] > [Parallel Time: 8,0 ms, GC Workers: 10] > [GC Worker Start (ms): Min: 61129248,9, Avg: 61129249,0, Max: > 61129249,0, Diff: 0,1] > [Ext Root Scanning (ms): Min: 0,4, Avg: 0,7, Max: 2,0, Diff: 1,6, > Sum: 7,4] > [Update RS (ms): Min: 0,0, Avg: 0,9, Max: 1,3, Diff: 1,3, Sum: 8,7] > [Processed Buffers: Min: 0, Avg: 14,6, Max: 38, Diff: 38, Sum: > 146] > [Scan RS (ms): Min: 0,1, Avg: 0,4, Max: 0,6, Diff: 0,5, Sum: 3,9] > [Code Root Scanning (ms): Min: 0,0, Avg: 0,0, Max: 0,0, Diff: 0,0, > Sum: 0,2] > [Object Copy (ms): Min: 5,7, Avg: 5,7, Max: 5,8, Diff: 0,1, Sum: > 57,5] > [Termination (ms): Min: 0,0, Avg: 0,1, Max: 0,1, Diff: 0,1, Sum: 0,7] > [GC Worker Other (ms): Min: 0,0, Avg: 0,0, Max: 0,0, Diff: 0,0, Sum: > 0,2] > [GC Worker Total (ms): Min: 7,8, Avg: 7,9, Max: 7,9, Diff: 0,1, Sum: > 78,6] > [GC Worker End (ms): Min: 61129256,8, Avg: 61129256,8, Max: > 61129256,9, Diff: 0,0] > [Code Root Fixup: 0,1 ms] > [Code Root Migration: 0,2 ms] > [Clear CT: 2,0 ms] > [Other: 3,1 ms] > [Choose CSet: 0,0 ms] > [Ref Proc: 1,6 ms] > [Ref Enq: 0,1 ms] > [Free CSet: 1,3 ms] > [Eden: 19,1G(19,1G)->0,0B(19,1G) Survivors: 64,0M->80,0M Heap: > 25,3G(32,0G)->6337,0M(32,0G)] > [Times: user=0,10 sys=0,01, real=0,01 secs] > > I unfortunately don't have the gc logs from when we were using cms, only > entries in the es logs. > > [gc][old][420117][9522] duration [46.2s], collections [2]/[47s], total > [46.2s]/[16.8m], memory [31.6gb]->[30.3gb]/[31.9gb], all_pools {[young] > [490.1mb]->[56.6mb]/[665.6mb]}{[survivor] [83.1mb]->[0b]/[83.1mb]}{[old] > [31gb]->[30.3gb]/[31.1gb]} > > and this never reclaimed as much space as it does now. > > > > On Thu, Sep 11, 2014 at 9:16 AM, [email protected] < > [email protected]> wrote: > >> Java 8 / G1GC works well here, what issues do you have? >> >> Jörg >> >> On Wed, Sep 10, 2014 at 8:13 PM, Robert Gardam <[email protected]> >> wrote: >> >>> I had been hitting my head up against heap issues until this afternoon >>> after enabling G1GC. >>> >>> What are the known issues with this type of GC? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/5e158da0-fd01-4dd1-8483-cf2671c675b2%40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/5e158da0-fd01-4dd1-8483-cf2671c675b2%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/6gum8WWaqsM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBvjnVve4V1gbzuULR%2BYtb4OD3AgukxfX1q_6wDg%3DOYg%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBvjnVve4V1gbzuULR%2BYtb4OD3AgukxfX1q_6wDg%3DOYg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAN%3D%2BWfpUvHnv71Wzh33KjejQmuPdr8md90Jn_OMTGdMeAeZ4YQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAN%3D%2BWfpUvHnv71Wzh33KjejQmuPdr8md90Jn_OMTGdMeAeZ4YQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGxG57ysggU%3DUfXhHz7eCgHR0cwhDpEwz7y29sQyMX3Lw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
