[JIRA] (JENKINS-51568) Pipeline jobs hanging in Build Executor even if it is finished

[brain...@yandex.ru (JIRA)]

Title: Message Title

Alexander Moiseenko commented on JENKINS-51568   Re: Pipeline jobs hanging in Build Executor even if it is finished I think we've found root cause of a problem. Broken `sleep` and `timeout` methods and executor hanging all relates to jenkins.util.Timer threads.  In hang state we cojvisualvm and watch threads state like this: And in normal case:   In thread-dump.txt we could find find jenkins.util.Timer stack traces, in our case the root cause was logfilesizechecker plugin, https://github.com/jenkinsci/logfilesizechecker-plugin/blob/master/src/main/java/hudson/plugins/logfilesizechecker/LogfilesizecheckerWrapper.java#L78 that uses timer every second to check log size and this operation creates additional cpu load    "jenkins.util.Timer [#9]" - Thread t@267 java.lang.Thread.State: RUNNABLE at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655) at java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:682) at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:781) at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546) at jenkins.model.Nodes.getNode(Nodes.java:295) at jenkins.model.Jenkins.getNode(Jenkins.java:2058) at hudson.model.Computer.getNode(Computer.java:590) at hudson.slaves.SlaveComputer.getNode(SlaveComputer.java:199) at hudson.slaves.SlaveComputer.getNode(SlaveComputer.java:96) at jenkins.model.Jenkins$7.compare(Jenkins.java:1927) at jenkins.model.Jenkins$7.compare(Jenkins.java:1925) at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360) at java.util.TimSort.sort(TimSort.java:234) at java.util.Arrays.sort(Arrays.java:1438) at jenkins.model.Jenkins.getComputers(Jenkins.java:1925) at hudson.model.Executor.of(Executor.java:941) at hudson.model.Run.getExecutor(Run.java:530) at hudson.plugins.logfilesizechecker.LogfilesizecheckerWrapper$LogSizeTimerTask.doRun(LogfilesizecheckerWrapper.java:107) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - locked <4dd03582> (a java.util.concurrent.ThreadPoolExecutor$Worker)   Our jenkins.util.Timer threads are mostly in park state and sleep works fine, since we've changed DELAY value from 1 second to 10 in https://github.com/jenkinsci/logfilesizechecker-plugin/blob/master/src/main/java/hudson/plugins/logfilesizechecker/LogfilesizecheckerWrapper.java#L46     Add Comment   This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.