[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Oleg Nenashev closed an issue as Cannot Reproduce No response from the requester Jenkins / JENKINS-53867 Too many open files caused Jenkins crashed Change By: Oleg Nenashev Status: Open Closed Resolution: Cannot Reproduce Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Oleg Nenashev commented on JENKINS-53867 Re: Too many open files caused Jenkins crashed Tapio Reijonen ping Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Oleg Nenashev commented on JENKINS-53867 Re: Too many open files caused Jenkins crashed l-wx-- 1 jenkins jenkins 64 Oct 9 01:12 1433 -> /jenkins/jobs/MasterOpenFiles1/builds/3/log and after an hour,... lrwx-- 1 jenkins jenkins 64 Oct 9 15:16 1433 -> socket:[75629928] It means that the file descriptor is reused for another operation. No evidence of the file leak With the current information it is impossible to diagnose the issue, unfortunately. If the issue still happens, please provide a reproducible scenario (e.g. as a Docker image) or configure your Jenkins instance to run with File Leak Detector or another similar tool Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen updated an issue Jenkins / JENKINS-53867 Too many open files caused Jenkins crashed Change By: Tapio Reijonen Priority: Critical Blocker Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen commented on JENKINS-53867 Re: Too many open files caused Jenkins crashed More info related this issue. I create a job, which list all fd's (socket handles created by jenkis). After couple run find out that those liked TCP fd's are related of the logging job output of the jenkins, I think. First the fd 1471 is for /jenkins/jobs/xxx-main-pull-request/builds/532/log l-wx-- 1 jenkins jenkins 64 Oct 9 13:08 1471 -> /jenkins/jobs/xxx-main-pull-request/builds/532/log And after some time (a hour), the same fd is still open(availabl), but it does not point anywhere anymore. lrwx-- 1 jenkins jenkins 64 Oct 9 13:08 1471 -> socket:[75632581] Here is an another example, btw this is the same job that I create to list these fd's: l-wx-- 1 jenkins jenkins 64 Oct 9 01:12 1433 -> /jenkins/jobs/MasterOpenFiles1/builds/3/log and after an hour,... lrwx-- 1 jenkins jenkins 64 Oct 9 15:16 1433 -> socket:[75629928] This job is very basic free style job,call linux shell script, don't use any of the additional plugins. pid=$(pgrep -u jenkins java) ls -lt /proc/$pid/fd Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Oleg Nenashev commented on JENKINS-53867 Re: Too many open files caused Jenkins crashed Well, using debug/profiling tools is the most simple way. But yes, they usually require running with JDK. Other ways you may use: Sniff the network, find out the connection destination. It may help to understand what invokes the connection Bisect the upgrade list. Try combinations of plugins and core until you find the cause of the leak Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen edited a comment on JENKINS-53867 Re: Too many open files caused Jenkins crashed I think, we can't run this "file-leak-detecor",java.lang.Error: Failed to activate file leak detector: Exception in thread "main" java.lang.IllegalStateException: Unable to find tools.jar at /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64/jre/../lib/tools.jar --- — you need to run this tool with a JDK at org.kohsuke.file_leak_detector.Main.loadAttachApi(Main.java:73) at org.kohsuke.file_leak_detector.Main.run(Main.java:48) at org.kohsuke.file_leak_detector.Main.main(Main.java:36) Is this the only way to find out which component cause the problem)? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen commented on JENKINS-53867 Re: Too many open files caused Jenkins crashed I think, we can't run this "file-leak-detecor", java.lang.Error: Failed to activate file leak detector: Exception in thread "main" java.lang.IllegalStateException: Unable to find tools.jar at /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-3.b12.el7_3.x86_64/jre/../lib/tools.jar — you need to run this tool with a JDK at org.kohsuke.file_leak_detector.Main.loadAttachApi(Main.java:73) at org.kohsuke.file_leak_detector.Main.run(Main.java:48) at org.kohsuke.file_leak_detector.Main.main(Main.java:36) Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Oleg Nenashev commented on JENKINS-53867 Re: Too many open files caused Jenkins crashed http://file-leak-detector.kohsuke.org may help, there is a plugin for it Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen edited a comment on JENKINS-53867 Re: Too many open files caused Jenkins crashed Unable to add support bundle because the size is too big. What info is required in bundle to find out what cause the TCP socket leak?Add attached jenkins2_plugins_2018-09-28.txt contains list of the used plugins. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen edited a comment on JENKINS-53867 Re: Too many open files caused Jenkins crashed Unable to add support bundle because the size is too big. What info is required in bundle to find out what cause the TCP socket leak? Add attached jenkins2_plugins_2018-09-28.txt contains list of the used plugins. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen updated an issue Jenkins / JENKINS-53867 Too many open files caused Jenkins crashed Change By: Tapio Reijonen Attachment: jenkins2_plugins_2018-09-28.txt Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title
Tapio Reijonen updated an issue
Jenkins / JENKINS-53867
Too many open files caused Jenkins crashed
Change By:
Tapio Reijonen
When we upgrade Jenkins core from the {color:#252424}2.73.3 to 2.89.4{color} and {color:#252424}2.89.4{color} to {color:#252424}2.107.3{color} the Jenkins start to leak file handles (also used plugins was updated) , see attached pictures created by Monitoring pluging. The linux lsof command shows that Jenkins leaks TCP sockets:{code:java}...java29737 jenkins 1244u sock0,7 0t0 24524616 protocol: TCPjava29737 jenkins 1245u sock0,7 0t0 17683939 protocol: TCPjava29737 jenkins 1246u sock0,7 0t0 95469787 protocol: TCPjava29737 jenkins 1247u sock0,7 0t0 90579011 protocol: TCPjava29737 jenkins 1248u sock0,7 0t0 33329570 protocol: TCP...{code}We can't use Open File Handle plugin to detect who leaks handles because we don't use required JDK (getting oops).
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options,
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen commented on JENKINS-53867 Re: Too many open files caused Jenkins crashed Unable to add support bundle because the size is too big. What info is required in bundle to find out what cause the TCP socket leak? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title
Tapio Reijonen updated an issue
Jenkins / JENKINS-53867
Too many open files caused Jenkins crashed
Change By:
Tapio Reijonen
When we upgrade Jenkins core from the {color:#252424}2.73.3 to 2.89.4{color} and {color:#252424}2.89.4{color} to {color:#252424}2.107.3{color} the Jenkins start to leak file handles, see attached pictures created by Monitoring pluging. The linux lsof command shows that Jenkins leaks TCP sockets:{code:java}...java29737 jenkins 1244u sock0,7 0t0 24524616 protocol: TCPjava29737 jenkins 1245u sock0,7 0t0 17683939 protocol: TCPjava29737 jenkins 1246u sock0,7 0t0 95469787 protocol: TCPjava29737 jenkins 1247u sock0,7 0t0 90579011 protocol: TCPjava29737 jenkins 1248u sock0,7 0t0 33329570 protocol: TCP...{code}We can't use Open File Handle plugin to detect who leaks handles because we don't use required JDK (getting oops).
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen updated an issue Jenkins / JENKINS-53867 Too many open files caused Jenkins crashed Change By: Tapio Reijonen Attachment: Jenkins-2.107.3-leaks.png Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen updated an issue Jenkins / JENKINS-53867 Too many open files caused Jenkins crashed Change By: Tapio Reijonen Attachment: Jenkins-2.89.4-leaks.png Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen updated an issue Jenkins / JENKINS-53867 Too many open files caused Jenkins crashed Change By: Tapio Reijonen Attachment: Jenkins-2.73.3-no-leaks.png Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-53867) Too many open files caused Jenkins crashed
Title: Message Title Tapio Reijonen created an issue Jenkins / JENKINS-53867 Too many open files caused Jenkins crashed Issue Type: Bug Assignee: Oleg Nenashev Components: _unsorted Created: 2018-10-02 06:19 Priority: Critical Reporter: Tapio Reijonen When we upgrade Jenkins core from the 2.73.3 to 2.89.4 and 2.89.4 to 2.107.3 the Jenkins start to leak file handles, see attached Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
