if it is the same instance with the agent disconnection errors, your problem is the NFS performance, it blocks tons of threads in IO operations that make agents cannot disconnect or connects blocking thousands of threads, the Jenkins instance works until is collapsed by the number of threads blocked or the amount of memory used by them. As I said in the other email thread rid of the NFS and your problems go away, if it is not possible, make that your NFS performs like a hard drive (+150MB/s) with more than 100 concurrent IO operations on small files for a period of 5-10 min, if you cannot make it that NFS will not work properly with Jenkins.
El lunes, 5 de agosto de 2019, 11:08:27 (UTC+2), Sverre Moe escribió: > > Have not yet resolved the issue. Not found a solution, nor what the cause > actually is. > > lørdag 3. august 2019 13.04.10 UTC+2 skrev guerkan demirci følgende: >> >> Hi Sverre Moe, >> >> could you resolve the issue? >> I have the same problem with Jenkins. >> >> Best >> >> >> Am Montag, 29. Juli 2019 11:30:58 UTC+2 schrieb Sverre Moe: >>> >>> I have analyzed the GC log >>> >>> >>> https://www.gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjgvLS1nYy0yMDE5LTA3LTI2XzE3LTMxLTIyLmxvZy4wLmN1cnJlbnQtLTE1LTI0LTQy&channel=WEB >>> >>> This may be related to another issue I am having when build agents are >>> getting disconnected. >>> https://groups.google.com/forum/#!topic/jenkinsci-users/UDyaH-kfqqA >>> >>> I also have an jstack output analyzed: >>> >>> https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTkvMDcvMjkvLS1qc3RhY2sudHh0LS05LTE2LTI3 >>> >>> søndag 14. juli 2019 13.48.57 UTC+2 skrev Sverre Moe følgende: >>>> >>>> I checked out that link >>>> >>>> Tried the ulimit settings >>>> >>>> https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#bulimitsettingsjustforlinuxos >>>> >>>> Will try the additional JVM flags suggested in >>>> >>>> https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support#ajavaparameters >>>> >>>> søndag 14. juli 2019 13.29.22 UTC+2 skrev Sverre Moe følgende: >>>>> >>>>> jenkins@meoscorebs12:~> ulimit -a >>>>> core file size (blocks, -c) unlimited >>>>> data seg size (kbytes, -d) unlimited >>>>> scheduling priority (-e) 0 >>>>> file size (blocks, -f) unlimited >>>>> pending signals (-i) 80229 >>>>> max locked memory (kbytes, -l) 64 >>>>> max memory size (kbytes, -m) unlimited >>>>> open files (-n) 1024 >>>>> pipe size (512 bytes, -p) 8 >>>>> POSIX message queues (bytes, -q) 819200 >>>>> real-time priority (-r) 0 >>>>> stack size (kbytes, -s) 8192 >>>>> cpu time (seconds, -t) unlimited >>>>> max user processes (-u) 80229 >>>>> virtual memory (kbytes, -v) unlimited >>>>> file locks (-x) unlimited >>>>> >>>>> >>>>> I have check the number of processes, with "ps aux", but not the >>>>> number of threads. >>>>> Right now, (Jenkins was restarted 2 day ago and few builds have run), >>>>> there are 264 threads for Jenkins currently. >>>>> Is there any way I can find out what each thread is for? >>>>> >>>>> lørdag 13. juli 2019 13.02.15 UTC+2 skrev Ivan Fernandez Calvo >>>>> følgende: >>>>>> >>>>>> Hi, >>>>>> >>>>>> When the issue happens, Did you check the number of threads that >>>>>> Jenkins has open? How many file descriptors can your process open (run >>>>>> ulimit >>>>>> -a with the user jenkins)? here you have a good KB about memory and >>>>>> user limit on Jenkins Prepare Jenkins for Support >>>>>> <https://support.cloudbees.com/hc/en-us/articles/222446987-Prepare-Jenkins-for-Support> >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8659bf13-1111-4076-a2c3-32e07d564aca%40googlegroups.com.
