I have not read the whole thread in detail, but the “Unable to create new native thread” OutOfMemoryErrors from your original thread where one of the stack traces involves org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing looks like it could be related to https://issues.jenkins-ci.org/browse/JENKINS-58684 <https://issues.jenkins-ci.org/browse/JENKINS-58684>, which is a thread leak caused by the SSE Gateway Plugin. You could try reverting the SSE Gateway Plugin to version 1.17 to see if that helps, although that might reintroduce a different, somewhat rarer memory leak (https://issues.jenkins-ci.org/browse/JENKINS-51057 <https://issues.jenkins-ci.org/browse/JENKINS-51057>). To test my hypothesis, if you are running SSE Gateway Plugin version 1.19, you can collect thread dumps over time and see if you seem to have a large number of threads named “EventDispatcher.retryProcessor” (unfortunately in version 1.18 and below the threads are automatically named “Timer #n”, which is less useful), which would confirm that you are hitting JENKINS-58684 <https://issues.jenkins-ci.org/browse/JENKINS-58684>.
The advice to stop building on master is definitely a good idea as well. > On Aug 14, 2019, at 07:11, Sverre Moe <[email protected]> wrote: > > We got an 30 minute free CloudBees support. It was too short to dig deeper to > find the problem, but the person I was talking to (after examining our logs) > mentioned what he thought was the problem and gave a suggestion. > > We should not use Jenkins master at all for builds (allocated with the > node("master") step). We had 15 Executors for Jenkins master. > > We could also try to Increase limits of hard nofile and nproc for jenkins > user, but the main recomondation was to remove all Executors for Jenkins > master. > > /etc/security/limits.conf > jenkins soft core unlimited > jenkins hard core unlimited > jenkins soft fsize unlimited > jenkins hard fsize unlimited > jenkins soft nofile 4096 > jenkins hard nofile 10240 #Was 8192 > jenkins soft nproc 30654 > jenkins hard nproc 60654 #Was 30654 > > > To remove Jenkins master Executors will take some time. We use Jenkins master > when we publish our build artifacts RPMs to our NFS file storage. > Since our RPM NFS is only attached to the Jenkins master it is not possible > at the moment. Unless we can use any other agent, then do a SCP onto our > Jenkins master with the RPM artifacts. > > > We had a few other circumstances where we used Jenkins master. Like checking > out a file to determine which build agent to actually use. These I have > already changed to use any available build agent instead. > > tirsdag 6. august 2019 09.48.50 UTC+2 skrev Sverre Moe følgende: > Sadly I was mistaken. We do not use NFS for JENKINS_HOME. > > We do however use NFS for the location where builds copy the RPM build > artifacts. > > mandag 5. august 2019 22.17.46 UTC+2 skrev Ivan Fernandez Calvo følgende: > Hi, > > Severe has another email thread open, I think it is the same Jenkins instance > https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer > > <https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer>. > I dunno what happens on your instance but probably it isn’t better that you > open another email thread with the description of your issue > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com > > <https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9FACF73D-081E-4DA8-92F0-86027EAC0868%40cloudbees.com.
