I was mistaken. We did not use NFS. The disk for JENKINS_HOME (Jenkins running on VM), is a LVM disk.
mandag 29. juli 2019 18.15.20 UTC+2 skrev Ivan Fernandez Calvo følgende: > > check the Cloudbees links, I've helped to write those KB when I was on > CloudBees :), I'm pretty sure that the NFS is your pain and the root cause > of all your problems if you can rid of it better. > > El lunes, 29 de julio de 2019, 18:03:59 (UTC+2), slide escribió: >> >> CloudBees (not my employer) has some resources on using NFS (generally >> the recommendation is to NOT use NFS for JENKINS_HOME). >> >> >> https://support.cloudbees.com/hc/en-us/articles/115000486312-CJP-Performance-Best-Practices-for-Linux#nfs >> >> and >> https://support.cloudbees.com/hc/en-us/articles/217479948-NFS-Guide >> >> On Mon, Jul 29, 2019 at 8:51 AM Sverre Moe <[email protected]> wrote: >> >>> Yes, we are using NFS for JENKINS_HOME. >>> >>> mandag 29. juli 2019 15.41.00 UTC+2 skrev Ivan Fernandez Calvo følgende: >>>> >>>> you have 83 threads in state:IN_NATIVE, probably stuck in IO >>>> operations, those 83 threads are blocking the other 382 threads, if you >>>> use >>>> an NFS or similar device for you Jenkins HOME this is probably your >>>> bottleneck, if not check the IO stats on the OS to see where you have the >>>> bottleneck. >>>> >>>> El lunes, 29 de julio de 2019, 11:20:50 (UTC+2), Sverre Moe escribió: >>>>> >>>>> I was unable to determine something from the stack output >>>>> Here is the result: >>>>> https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3 >>>>> >>>>> torsdag 18. juli 2019 11.28.06 UTC+2 skrev Sverre Moe følgende: >>>>>> >>>>>> There is no such reference in my jstack output. >>>>>> The output says no deadlock detected. >>>>>> I will try that site for analyzing the jstack. >>>>>> >>>>>> Even a normal running Jenkins has many BLOCKED threads. If that is >>>>>> normal I don't know. >>>>>> >>>>>> We have a test Jenkins instance running on Java 11. That one does not >>>>>> have any BLOCKED threads. >>>>>> Our production Jenkins is running Java 8u181. >>>>>> >>>>>> torsdag 18. juli 2019 11.04.16 UTC+2 skrev Ivan Fernandez Calvo >>>>>> følgende: >>>>>>> >>>>>>> In that dump I can not see which thread is blocking the others, the >>>>>>> jstack output has a reference on each thread that said what thread is >>>>>>> the >>>>>>> blocker on each thread (- locked <0x00000000> a java.lang.Object), >>>>>>> you can try to analyze those thread dump with >>>>>>> https://fastthread.io/index.jsp or other online tools to see if you >>>>>>> see something relevant, it looks like there is a deadlock. >>>>>>> >>>>>>> https://dzone.com/articles/how-to-read-a-thread-dump >>>>>>> >>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/cc2d0bdb-b15f-4bec-a0a3-0562ea8c7df7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Website: http://earl-of-code.com >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/da7baa40-bdac-490b-bad6-067aba4591b4%40googlegroups.com.
