We do have one NFS, for copying build artifacts to RPM repository. tirsdag 6. august 2019 09.08.48 UTC+2 skrev Sverre Moe følgende: > > I was mistaken. We did not use NFS. > The disk for JENKINS_HOME (Jenkins running on VM), is a LVM disk. > > mandag 29. juli 2019 18.15.20 UTC+2 skrev Ivan Fernandez Calvo følgende: >> >> check the Cloudbees links, I've helped to write those KB when I was on >> CloudBees :), I'm pretty sure that the NFS is your pain and the root cause >> of all your problems if you can rid of it better. >> >> El lunes, 29 de julio de 2019, 18:03:59 (UTC+2), slide escribió: >>> >>> CloudBees (not my employer) has some resources on using NFS (generally >>> the recommendation is to NOT use NFS for JENKINS_HOME). >>> >>> >>> https://support.cloudbees.com/hc/en-us/articles/115000486312-CJP-Performance-Best-Practices-for-Linux#nfs >>> >>> and >>> https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide >>> >>> On Mon, Jul 29, 2019 at 8:51 AM Sverre Moe <[email protected]> wrote: >>> >>>> Yes, we are using NFS for JENKINS_HOME. >>>> >>>> mandag 29. juli 2019 15.41.00 UTC+2 skrev Ivan Fernandez Calvo følgende: >>>>> >>>>> you have 83 threads in state:IN_NATIVE, probably stuck in IO >>>>> operations, those 83 threads are blocking the other 382 threads, if you >>>>> use >>>>> an NFS or similar device for you Jenkins HOME this is probably your >>>>> bottleneck, if not check the IO stats on the OS to see where you have the >>>>> bottleneck. >>>>> >>>>> El lunes, 29 de julio de 2019, 11:20:50 (UTC+2), Sverre Moe escribió: >>>>>> >>>>>> I was unable to determine something from the stack output >>>>>> Here is the result: >>>>>> https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3 >>>>>> >>>>>> torsdag 18. juli 2019 11.28.06 UTC+2 skrev Sverre Moe følgende: >>>>>>> >>>>>>> There is no such reference in my jstack output. >>>>>>> The output says no deadlock detected. >>>>>>> I will try that site for analyzing the jstack. >>>>>>> >>>>>>> Even a normal running Jenkins has many BLOCKED threads. If that is >>>>>>> normal I don't know. >>>>>>> >>>>>>> We have a test Jenkins instance running on Java 11. That one does >>>>>>> not have any BLOCKED threads. >>>>>>> Our production Jenkins is running Java 8u181. >>>>>>> >>>>>>> torsdag 18. juli 2019 11.04.16 UTC+2 skrev Ivan Fernandez Calvo >>>>>>> følgende: >>>>>>>> >>>>>>>> In that dump I can not see which thread is blocking the others, the >>>>>>>> jstack output has a reference on each thread that said what thread is >>>>>>>> the >>>>>>>> blocker on each thread (- locked <0x00000000> a java.lang.Object), >>>>>>>> you can try to analyze those thread dump with >>>>>>>> https://fastthread.io/index.jsp or other online tools to see if >>>>>>>> you see something relevant, it looks like there is a deadlock. >>>>>>>> >>>>>>>> https://dzone.com/articles/how-to-read-a-thread-dump >>>>>>>> >>>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Website: http://earl-of-code.com >>> >>
-- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d6699e75-4a3c-4125-8b5d-4df0fc12fb47%40googlegroups.com.
