I only have the option to downgrade to 1.18 of Server Sent Events (SSE) Gateway Plugin I would have to download the 1.17 and manually downgrade it.
>From the discussions it seems I also need to downgrade BlueOcean to 1.17 > Downgrading to BlueOcean 1.17 (which in turn uses sse-gateway 1.17) appears to have resolved our issue This would be much more work. I would need to install all the BlueOcean 1.17 manually as I can only downgrade to 1.18.0 I might be willing to try this, even with the risk of https://issues.jenkins-ci.org/browse/JENKINS-51057 I am running SSE 1.19, and have previously recorded jstack from Jenkins PID. I could not find any EventDispatcher.retryProcessor \t onsdag 14. august 2019 15.38.17 UTC+2 skrev Devin Nusbaum følgende: > > I have not read the whole thread in detail, but the “Unable to create new > native thread” OutOfMemoryErrors from your original thread where one of the > stack traces involves > org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing > looks > like it could be related to > https://issues.jenkins-ci.org/browse/JENKINS-58684, which is a thread > leak caused by the SSE Gateway Plugin. You could try reverting the SSE > Gateway Plugin to version 1.17 to see if that helps, although that might > reintroduce a different, somewhat rarer memory leak ( > https://issues.jenkins-ci.org/browse/JENKINS-51057). To test my > hypothesis, if you are running SSE Gateway Plugin version 1.19, you can > collect thread dumps over time and see if you seem to have a large number > of threads named “EventDispatcher.retryProcessor” (unfortunately in version > 1.18 and below the threads are automatically named “Timer #n”, which is > less useful), which would confirm that you are hitting JENKINS-58684 > <https://issues.jenkins-ci.org/browse/JENKINS-58684>. > > The advice to stop building on master is definitely a good idea as well. > > On Aug 14, 2019, at 07:11, Sverre Moe <[email protected] <javascript:>> > wrote: > > We got an 30 minute free CloudBees support. It was too short to dig deeper > to find the problem, but the person I was talking to (after examining our > logs) mentioned what he thought was the problem and gave a suggestion. > > We should not use Jenkins master at all for builds (allocated with the > node("master") step). We had 15 Executors for Jenkins master. > > We could also try to Increase limits of hard nofile and nproc for jenkins > user, but the main recomondation was to remove all Executors for Jenkins > master. > > /etc/security/limits.conf > jenkins soft core unlimited > jenkins hard core unlimited > jenkins soft fsize unlimited > jenkins hard fsize unlimited > jenkins soft nofile 4096 > jenkins hard nofile 10240 #Was 8192 > jenkins soft nproc 30654 > jenkins hard nproc 60654 #Was 30654 > > > To remove Jenkins master Executors will take some time. We use Jenkins > master when we publish our build artifacts RPMs to our NFS file storage. > Since our RPM NFS is only attached to the Jenkins master it is not > possible at the moment. Unless we can use any other agent, then do a SCP > onto our Jenkins master with the RPM artifacts. > > > We had a few other circumstances where we used Jenkins master. Like > checking out a file to determine which build agent to actually use. These I > have already changed to use any available build agent instead. > > tirsdag 6. august 2019 09.48.50 UTC+2 skrev Sverre Moe følgende: >> >> Sadly I was mistaken. We do not use NFS for JENKINS_HOME. >> >> We do however use NFS for the location where builds copy the RPM build >> artifacts. >> >> mandag 5. august 2019 22.17.46 UTC+2 skrev Ivan Fernandez Calvo følgende: >>> >>> Hi, >>> >>> Severe has another email thread open, I think it is the same Jenkins >>> instance >>> https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer. >>> >>> I dunno what happens on your instance but probably it isn’t better that you >>> open another email thread with the description of your issue >> >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com > > <https://groups.google.com/d/msgid/jenkinsci-users/3e728790-b2f5-4ae1-a9fe-512a5c912d61%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d026b611-fab1-420a-a870-de5e6df291cf%40googlegroups.com.
