Re: Setting CMS or G1 as the default GC algorithm for Jenkins?

[Samuel Van Oort]

For interested parties, this is tracked 
in https://issues.jenkins-ci.org/browse/JENKINS-38802

On Tuesday, October 11, 2016 at 12:57:49 PM UTC-4, Samuel Van Oort wrote:
>
> Baptiste, I completely agree -- GC logs are a gimme for any production 
> setting IMO (and since it's in the JIRA, it won't be missed).
>
> As a point of reference, I'd reached out to my friends (and former 
> colleagues) at a multi-billion dollar company known for open source, and 
> CMS is still the preference there for production app servers.  It kept 
> request times stable on the business-critical services layer I owned, and 
> prevented errors/hangups due to timeouts from GC pauses.
>
> On Saturday, October 8, 2016 at 5:06:16 AM UTC-4, Baptiste Mathus wrote:
>>
>> +1. On the mid-size cluster I worked on in the past we did that quite a 
>> long time ago and it did improve the user experience. Clearly less pauses.
>>
>> (As said on Jira, while doing so I would definitely take the opportunity 
>> to also set up gc logs enabled by default (with rotation). There's no 
>> noticeable impact on perf even under high load, and this is invaluable 
>> information when things go wrong.)
>>
>> Le 7 oct. 2016 9:45 PM, "Samuel Van Oort"  a 
>> écrit :
>>
>>> Kanstanstin, 
>>>
>>> > suggest collect GC monitoring 
>>>
>>> I'm glad you asked! I actually spent yesterday digging into problems 
>>> with two high-load, large Jenkins instances which were experiencing issues, 
>>> and for which we had GC logs available.  In both cases, we observed 
>>> multi-second pause-the-world full GC cycles with the default parallel GC -- 
>>> please find attached a screenshot from one of the GC analyses.
>>>
>>> > G1 as default for java9... depends on how much patches and differences 
>>> are between java8 and java9 and especially oracle vs openjdk.
>>>
>>> Indeed.  It's pretty contentious for Java 9 as well.
>>>
>>> The benchmarks I've seen for G1 on JDK 7 have been consistently poor, so 
>>> best to avoid it.
>>>
>>> On Java 8, where G1 has gotten a lot of optimizations over time, it 
>>> seems to trade back and forth with CMS if you care about pause time, 
>>> depending on situation:
>>> http://eivindw.github.io/2016/01/08/comparing-gc-collectors.html
>>> http://blog.dripstat.com/java-g1-gc-review/
>>>
>>> CMS is harder to tune, and falls back to longer stop-the-world pauses 
>>> eventually in some cases, but tends to chew a bit less CPU.  
>>> G1 seems to have an advantage at >4 GB heaps based on benchmarks and 
>>> tuning advice I've seen (though I haven't run it in a production setting 
>>> yet). 
>>>
>>> I think for *most* users running CI/CD systems with generally-spiky 
>>> load, generally very low CPU load (or at least some unloaded cores), and 
>>> multicore systems with 2-8 GB heap *probably* G1 is the best overall 
>>> solution.  CMS has some advantages if your system tends to run highly 
>>> loaded (better GC throughput). 
>>>
>>> Maybe you've got experiences to contribute though? 
>>>
>>> @jglick:
>>>  > if the user is selecting a newer JRE than you expected, it would be 
>>> better to let the JRE’s own defaults apply. 
>>>
>>> Point.  My expectation would be: java 7 --> CMS, Java 8 --> probably G1 
>>> (maybe CMS), Java 9+ --> default for platform. 
>>>
>>> On Friday, October 7, 2016 at 1:27:55 PM UTC-4, Kanstantsin Shautsou 
>>> wrote:

 Applying theory to practise could be not proved without facts. I would 
 suggest collect GC monitoring from jenkins on jenkins (or whatever it 
 called now) and compare. Then it will be absolutely clear what is better.

 G1 as default for java9... depends on how much patches and differences 
 are between java8 and java9 and especially oracle vs openjdk.

 On Friday, October 7, 2016 at 12:29:56 AM UTC+3, Samuel Van Oort wrote:
>
> Hi guys, 
> I'd like to propose that we explicitly set the Java args for the 
> Jenkins packages to use either Concurrent Mark Sweep or G1 as the default 
> GC algorithm. 
>
> The reason for this is that Jenkins is generally used as a 
> long-running server process, and large stop-the-world GC pauses can pose 
> a 
> problem for stability and user experience.   The default Java GC 
> algorithm 
> (ParallelGC) is tuned to optimize throughput at the expense of 
> potentially 
> multi-second major GC pauses with large heaps, which is obviously not a 
> good fit for this case. This is why Oracle is moving to G1 as the default 
> as of Java 9.
>
> I would suggest either CMS or G1 make good defaults for normal Jenkins 
> users, because they greatly reduce average and maximum GC pause times.
>
> Thoughts?
>
> Regards,
> Sam Van Oort
>
 -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Jenkins Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to jenkinsci-de...@goog

Re: Setting CMS or G1 as the default GC algorithm for Jenkins?

[Samuel Van Oort]

Baptiste, I completely agree -- GC logs are a gimme for any production 
setting IMO (and since it's in the JIRA, it won't be missed).

As a point of reference, I'd reached out to my friends (and former 
colleagues) at a multi-billion dollar company known for open source, and 
CMS is still the preference there for production app servers.  It kept 
request times stable on the business-critical services layer I owned, and 
prevented errors/hangups due to timeouts from GC pauses.

On Saturday, October 8, 2016 at 5:06:16 AM UTC-4, Baptiste Mathus wrote:
>
> +1. On the mid-size cluster I worked on in the past we did that quite a 
> long time ago and it did improve the user experience. Clearly less pauses.
>
> (As said on Jira, while doing so I would definitely take the opportunity 
> to also set up gc logs enabled by default (with rotation). There's no 
> noticeable impact on perf even under high load, and this is invaluable 
> information when things go wrong.)
>
> Le 7 oct. 2016 9:45 PM, "Samuel Van Oort"  > a écrit :
>
>> Kanstanstin, 
>>
>> > suggest collect GC monitoring 
>>
>> I'm glad you asked! I actually spent yesterday digging into problems with 
>> two high-load, large Jenkins instances which were experiencing issues, and 
>> for which we had GC logs available.  In both cases, we observed 
>> multi-second pause-the-world full GC cycles with the default parallel GC -- 
>> please find attached a screenshot from one of the GC analyses.
>>
>> > G1 as default for java9... depends on how much patches and differences 
>> are between java8 and java9 and especially oracle vs openjdk.
>>
>> Indeed.  It's pretty contentious for Java 9 as well.
>>
>> The benchmarks I've seen for G1 on JDK 7 have been consistently poor, so 
>> best to avoid it.
>>
>> On Java 8, where G1 has gotten a lot of optimizations over time, it seems 
>> to trade back and forth with CMS if you care about pause time, depending on 
>> situation:
>> http://eivindw.github.io/2016/01/08/comparing-gc-collectors.html
>> http://blog.dripstat.com/java-g1-gc-review/
>>
>> CMS is harder to tune, and falls back to longer stop-the-world pauses 
>> eventually in some cases, but tends to chew a bit less CPU.  
>> G1 seems to have an advantage at >4 GB heaps based on benchmarks and 
>> tuning advice I've seen (though I haven't run it in a production setting 
>> yet). 
>>
>> I think for *most* users running CI/CD systems with generally-spiky load, 
>> generally very low CPU load (or at least some unloaded cores), and 
>> multicore systems with 2-8 GB heap *probably* G1 is the best overall 
>> solution.  CMS has some advantages if your system tends to run highly 
>> loaded (better GC throughput). 
>>
>> Maybe you've got experiences to contribute though? 
>>
>> @jglick:
>>  > if the user is selecting a newer JRE than you expected, it would be 
>> better to let the JRE’s own defaults apply. 
>>
>> Point.  My expectation would be: java 7 --> CMS, Java 8 --> probably G1 
>> (maybe CMS), Java 9+ --> default for platform. 
>>
>> On Friday, October 7, 2016 at 1:27:55 PM UTC-4, Kanstantsin Shautsou 
>> wrote:
>>>
>>> Applying theory to practise could be not proved without facts. I would 
>>> suggest collect GC monitoring from jenkins on jenkins (or whatever it 
>>> called now) and compare. Then it will be absolutely clear what is better.
>>>
>>> G1 as default for java9... depends on how much patches and differences 
>>> are between java8 and java9 and especially oracle vs openjdk.
>>>
>>> On Friday, October 7, 2016 at 12:29:56 AM UTC+3, Samuel Van Oort wrote:

 Hi guys, 
 I'd like to propose that we explicitly set the Java args for the 
 Jenkins packages to use either Concurrent Mark Sweep or G1 as the default 
 GC algorithm. 

 The reason for this is that Jenkins is generally used as a long-running 
 server process, and large stop-the-world GC pauses can pose a problem for 
 stability and user experience.   The default Java GC algorithm 
 (ParallelGC) 
 is tuned to optimize throughput at the expense of potentially multi-second 
 major GC pauses with large heaps, which is obviously not a good fit for 
 this case. This is why Oracle is moving to G1 as the default as of Java 9.

 I would suggest either CMS or G1 make good defaults for normal Jenkins 
 users, because they greatly reduce average and maximum GC pause times.

 Thoughts?

 Regards,
 Sam Van Oort

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Jenkins Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jenkinsci-de...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/jenkinsci-dev/11b2a288-119f-4f8c-a87a-b0ce13b8aa26%40googlegroups.com
>>  
>> 

Re: Setting CMS or G1 as the default GC algorithm for Jenkins?

[Baptiste Mathus]

+1. On the mid-size cluster I worked on in the past we did that quite a
long time ago and it did improve the user experience. Clearly less pauses.

(As said on Jira, while doing so I would definitely take the opportunity to
also set up gc logs enabled by default (with rotation). There's no
noticeable impact on perf even under high load, and this is invaluable
information when things go wrong.)

Le 7 oct. 2016 9:45 PM, "Samuel Van Oort"  a écrit :

> Kanstanstin,
>
> > suggest collect GC monitoring
>
> I'm glad you asked! I actually spent yesterday digging into problems with
> two high-load, large Jenkins instances which were experiencing issues, and
> for which we had GC logs available.  In both cases, we observed
> multi-second pause-the-world full GC cycles with the default parallel GC --
> please find attached a screenshot from one of the GC analyses.
>
> > G1 as default for java9... depends on how much patches and differences
> are between java8 and java9 and especially oracle vs openjdk.
>
> Indeed.  It's pretty contentious for Java 9 as well.
>
> The benchmarks I've seen for G1 on JDK 7 have been consistently poor, so
> best to avoid it.
>
> On Java 8, where G1 has gotten a lot of optimizations over time, it seems
> to trade back and forth with CMS if you care about pause time, depending on
> situation:
> http://eivindw.github.io/2016/01/08/comparing-gc-collectors.html
> http://blog.dripstat.com/java-g1-gc-review/
>
> CMS is harder to tune, and falls back to longer stop-the-world pauses
> eventually in some cases, but tends to chew a bit less CPU.
> G1 seems to have an advantage at >4 GB heaps based on benchmarks and
> tuning advice I've seen (though I haven't run it in a production setting
> yet).
>
> I think for *most* users running CI/CD systems with generally-spiky load,
> generally very low CPU load (or at least some unloaded cores), and
> multicore systems with 2-8 GB heap *probably* G1 is the best overall
> solution.  CMS has some advantages if your system tends to run highly
> loaded (better GC throughput).
>
> Maybe you've got experiences to contribute though?
>
> @jglick:
>  > if the user is selecting a newer JRE than you expected, it would be
> better to let the JRE’s own defaults apply.
>
> Point.  My expectation would be: java 7 --> CMS, Java 8 --> probably G1
> (maybe CMS), Java 9+ --> default for platform.
>
> On Friday, October 7, 2016 at 1:27:55 PM UTC-4, Kanstantsin Shautsou wrote:
>>
>> Applying theory to practise could be not proved without facts. I would
>> suggest collect GC monitoring from jenkins on jenkins (or whatever it
>> called now) and compare. Then it will be absolutely clear what is better.
>>
>> G1 as default for java9... depends on how much patches and differences
>> are between java8 and java9 and especially oracle vs openjdk.
>>
>> On Friday, October 7, 2016 at 12:29:56 AM UTC+3, Samuel Van Oort wrote:
>>>
>>> Hi guys,
>>> I'd like to propose that we explicitly set the Java args for the Jenkins
>>> packages to use either Concurrent Mark Sweep or G1 as the default GC
>>> algorithm.
>>>
>>> The reason for this is that Jenkins is generally used as a long-running
>>> server process, and large stop-the-world GC pauses can pose a problem for
>>> stability and user experience.   The default Java GC algorithm (ParallelGC)
>>> is tuned to optimize throughput at the expense of potentially multi-second
>>> major GC pauses with large heaps, which is obviously not a good fit for
>>> this case. This is why Oracle is moving to G1 as the default as of Java 9.
>>>
>>> I would suggest either CMS or G1 make good defaults for normal Jenkins
>>> users, because they greatly reduce average and maximum GC pause times.
>>>
>>> Thoughts?
>>>
>>> Regards,
>>> Sam Van Oort
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jenkinsci-dev/11b2a288-119f-4f8c-a87a-b0ce13b8aa26%
> 40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS533n6Mjo5KRNSEFLg6vy1CX2PubKqhCup8JPD6Cgyu6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting CMS or G1 as the default GC algorithm for Jenkins?

[Samuel Van Oort]

Kanstanstin, 

> suggest collect GC monitoring 

I'm glad you asked! I actually spent yesterday digging into problems with 
two high-load, large Jenkins instances which were experiencing issues, and 
for which we had GC logs available.  In both cases, we observed 
multi-second pause-the-world full GC cycles with the default parallel GC -- 
please find attached a screenshot from one of the GC analyses.

> G1 as default for java9... depends on how much patches and differences 
are between java8 and java9 and especially oracle vs openjdk.

Indeed.  It's pretty contentious for Java 9 as well.

The benchmarks I've seen for G1 on JDK 7 have been consistently poor, so 
best to avoid it.

On Java 8, where G1 has gotten a lot of optimizations over time, it seems 
to trade back and forth with CMS if you care about pause time, depending on 
situation:
http://eivindw.github.io/2016/01/08/comparing-gc-collectors.html
http://blog.dripstat.com/java-g1-gc-review/

CMS is harder to tune, and falls back to longer stop-the-world pauses 
eventually in some cases, but tends to chew a bit less CPU.  
G1 seems to have an advantage at >4 GB heaps based on benchmarks and tuning 
advice I've seen (though I haven't run it in a production setting yet). 

I think for *most* users running CI/CD systems with generally-spiky load, 
generally very low CPU load (or at least some unloaded cores), and 
multicore systems with 2-8 GB heap *probably* G1 is the best overall 
solution.  CMS has some advantages if your system tends to run highly 
loaded (better GC throughput). 

Maybe you've got experiences to contribute though? 

@jglick:
 > if the user is selecting a newer JRE than you expected, it would be 
better to let the JRE’s own defaults apply. 

Point.  My expectation would be: java 7 --> CMS, Java 8 --> probably G1 
(maybe CMS), Java 9+ --> default for platform. 

On Friday, October 7, 2016 at 1:27:55 PM UTC-4, Kanstantsin Shautsou wrote:
>
> Applying theory to practise could be not proved without facts. I would 
> suggest collect GC monitoring from jenkins on jenkins (or whatever it 
> called now) and compare. Then it will be absolutely clear what is better.
>
> G1 as default for java9... depends on how much patches and differences are 
> between java8 and java9 and especially oracle vs openjdk.
>
> On Friday, October 7, 2016 at 12:29:56 AM UTC+3, Samuel Van Oort wrote:
>>
>> Hi guys, 
>> I'd like to propose that we explicitly set the Java args for the Jenkins 
>> packages to use either Concurrent Mark Sweep or G1 as the default GC 
>> algorithm. 
>>
>> The reason for this is that Jenkins is generally used as a long-running 
>> server process, and large stop-the-world GC pauses can pose a problem for 
>> stability and user experience.   The default Java GC algorithm (ParallelGC) 
>> is tuned to optimize throughput at the expense of potentially multi-second 
>> major GC pauses with large heaps, which is obviously not a good fit for 
>> this case. This is why Oracle is moving to G1 as the default as of Java 9.
>>
>> I would suggest either CMS or G1 make good defaults for normal Jenkins 
>> users, because they greatly reduce average and maximum GC pause times.
>>
>> Thoughts?
>>
>> Regards,
>> Sam Van Oort
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/11b2a288-119f-4f8c-a87a-b0ce13b8aa26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting CMS or G1 as the default GC algorithm for Jenkins?

[Jesse Glick]

On Thu, Oct 6, 2016 at 5:29 PM, Samuel Van Oort  wrote:
> I'd like to propose that we explicitly set the Java args for the Jenkins
> packages to use either Concurrent Mark Sweep or G1 as the default GC
> algorithm.

Fine insofar as that same package is pinning the Java version. But if
the user is selecting a newer JRE than you expected, it would be
better to let the JRE’s own defaults apply.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0iAdb_Z%2BH5Pyq-4kF__dNp7%2BCkWijgYMhmHWiDED0zJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting CMS or G1 as the default GC algorithm for Jenkins?

[Kanstantsin Shautsou]

Applying theory to practise could be not proved without facts. I would 
suggest collect GC monitoring from jenkins on jenkins (or whatever it 
called now) and compare. Then it will be absolutely clear what is better.

G1 as default for java9... depends on how much patches and differences are 
between java8 and java9 and especially oracle vs openjdk.

On Friday, October 7, 2016 at 12:29:56 AM UTC+3, Samuel Van Oort wrote:
>
> Hi guys, 
> I'd like to propose that we explicitly set the Java args for the Jenkins 
> packages to use either Concurrent Mark Sweep or G1 as the default GC 
> algorithm. 
>
> The reason for this is that Jenkins is generally used as a long-running 
> server process, and large stop-the-world GC pauses can pose a problem for 
> stability and user experience.   The default Java GC algorithm (ParallelGC) 
> is tuned to optimize throughput at the expense of potentially multi-second 
> major GC pauses with large heaps, which is obviously not a good fit for 
> this case. This is why Oracle is moving to G1 as the default as of Java 9.
>
> I would suggest either CMS or G1 make good defaults for normal Jenkins 
> users, because they greatly reduce average and maximum GC pause times.
>
> Thoughts?
>
> Regards,
> Sam Van Oort
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/3d14ad26-e3fe-4c1f-9a0c-29ff705ec2e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Setting CMS or G1 as the default GC algorithm for Jenkins?

[nicolas de loof]

+1

2016-10-06 23:29 GMT+02:00 Samuel Van Oort :

> Hi guys,
> I'd like to propose that we explicitly set the Java args for the Jenkins
> packages to use either Concurrent Mark Sweep or G1 as the default GC
> algorithm.
>
> The reason for this is that Jenkins is generally used as a long-running
> server process, and large stop-the-world GC pauses can pose a problem for
> stability and user experience.   The default Java GC algorithm (ParallelGC)
> is tuned to optimize throughput at the expense of potentially multi-second
> major GC pauses with large heaps, which is obviously not a good fit for
> this case. This is why Oracle is moving to G1 as the default as of Java 9.
>
> I would suggest either CMS or G1 make good defaults for normal Jenkins
> users, because they greatly reduce average and maximum GC pause times.
>
> Thoughts?
>
> Regards,
> Sam Van Oort
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/jenkinsci-dev/c009a299-1b23-4cb8-97ef-c6718fc279e1%
> 40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CANMVJzmBaccZgfd93-_pEomH6CVF93w5J5AZjpRuVoKqG0RpzA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.