Hey there - I think it might be worth posting the pipeline of your orchestration job that runs the 12000 builds for anyone to comment in more detail.
>From what I understand, if your pipeline is not orchestrating each build in a node section it will use something called a 'flyweight executor' on the Jenkins master. If this is happening, even though you have 100 agents your master may be heavily loaded running the pipeline - I guess each build allocates some Java objects that have to be GCed. If I can't find any documentation, I tend to go onto Github to look at the source code to understand what the Jenkins pipeline is doing behind the scenes: https://github.com/jenkinsci/workflow-cps-plugin. I ran a test on a system where I had an orchestrator kicking off 1000 builds every 4 minutes on a schedule. After a few days the Jenkins service stopped responding (Java memory issues). One thing I have found I needed to pay attention to was file and process limits as documented here: https://support.cloudbees.com/hc/en-us/articles/204231510-Memory-problem-unable-to-create-new-native-thread- All the best, --Bill On Thursday, 23 February 2017 21:33:42 UTC, Chris Overend wrote: > > So not sure if this is a Jenkins limitation or pipeline. > The jobs never exceeded available resources. > The garbage collection was stable. > > So why did it lock-up? > > It did say I used > > - 2950 million active threads > - 350 threads > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a504182f-ebf8-46a6-b343-03f639fefc8a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
