On 01/14/2017 11:53 AM, Baptiste Mathus wrote:
Bottom line being: not worth the engineering time to diagnose issues
when many big builds together at the wrong time will basically kill the
whole machine.
Better isolate with dedicated VMs per agent.
Each executor basically spins new VM via Vagrant/cloud provider - so
slaves are basically offloaded to dynamic VMs.
*BUT* - what's hurting me - biggest issue currently - is that at some
point Jenkins just gets "stuck".
What do I mean by stuck:
- jobs pile up in queue
- queue grows bigger and bigger
- there are 50-100 available executors but jobs from queue
either don't get started or get started after couple of hours...
I haven't pinpointed yet when or why exactly happens, but only solution
is to restart Jenkins master.
No. That is wrong. Curious where you heard that. I regularly see masters
with hundreds of agents.
I think Cloudbees, but it's information from third hand so I can't vouch
for it.
One thing that can arise on big configs is the GC misconfiguration (the
JVM defaults to parallel GC, which is basically a very unfortunate
choice for user facing apps like Jenkins masters, or any webapp really.
Only suited for batches...). For example resulting in long freezes from
a user standpoint.
We cannot unfortunately improve that where ppl are running the war
directly but there's work ongoing to choose better defaults for other
packagings (thanks Sam!)
See Stephen's talk, where basically he played with many definitions of
"the biggest cluster in the world" :)
Actually there weren't any major GC shitstorms that we had to endure...
--
You received this message because you are subscribed to the Google Groups "Jenkins
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/36ecbc35-f0bf-5cd3-72b7-d15fa6410949%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
