[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Damien Roche commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Our timeouts are set quite high, I'll reduce those today. It looks like this is known issue in the Ec2 Plugin - JENKINS-53858 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Damien Roche, we had similar issues before - and I fear that this is only loosely related to this issue here. In my case, we are not using EC2 cloud but the docker cloud plugin (YAD). Root cause for us was that the plugin did not have a connection timeout configured. If then a connection to the cloud manager fails, the thread is waiting eternally for an answer. For the sake of consistency, however, the thread aquired a lock, which then is never released... and all the blues started... That is why I would suggest you to have a look at your timeout values (I don't know whether they are configurable in the case of EC2) - and if applicable post them here for further cross-checking. If they are too high, you should fix that first. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Damien Roche commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" I took a thread dump during our encounter of this issue in the log today. We seem to encounter a Deadlock with Ec2 provisioning - possibly related to the changes introduced to spawn instances without delay. Have others enabled this feature? Our symptoms: Jenkins queue will grow (not substantially) - jobs don't seem to execute tho we have available executors. New jobs fail to even launch - nothing in logs either. All intensive purposes we don't get any lightweight executors. We see the following log entry multiple times for jobs that were in progress. Jan 02, 2019 10:02:36 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0 INFO: Running CpsFlowExecution[Owner[/master #59]] unresponsive for 55 min hudson.plugins.ec2.AmazonEC2Cloud is blocking 3 threads. jenkins.util.Timer [#7] thread obtained hudson.plugins.ec2.AmazonEC2Cloud's lock & did not release it. Due to that 3 threads are BLOCKED as shown in the below graph. If threads are BLOCKED for prolonged period, application will become unresponsive. Examine 'jenkins.util.Timer [#7]' stacktrace to see why lock is not released. jenkins.util.Timer [#7] - priority:5 - threadId:0x7f21a00f9800 - nativeId:0x1e3ac - nativeId (decimal):123820 - state:WAITING stackTrace: java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xc06ca768> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1438) at hudson.model.Queue.withLock(Queue.java:1301) at jenkins.model.Nodes.updateNode(Nodes.java:193) at jenkins.model.Jenkins.updateNode(Jenkins.java:2095) at hudson.model.Node.save(Node.java:140) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList.replaceBy(PersistedList.java:85) at hudson.model.Slave.(Slave.java:198) at hudson.plugins.ec2.EC2AbstractSlave.(EC2AbstractSlave.java:138) at hudson.plugins.ec2.EC2OndemandSlave.(EC2OndemandSlave.java:49) at hudson.plugins.ec2.EC2OndemandSlave.(EC2OndemandSlave.java:42) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:963) at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:660) at
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Damien Roche updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Damien Roche Attachment: Screen Shot 2019-01-02 at 22.46.07.png Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Joe Rioux commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" We are experiencing the same, and also have to reboot our server several times per day. Then sometimes it goes for days without a problem. I filed this bug - https://issues.jenkins-ci.org/browse/JENKINS-54894 We are going to try creating a new server to resolve this since we also have 2 other servers, configured nearly the same as the one we keep having to reboot. The new one will be identical to the 2 we don't have a problem on. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Ben Herfurth updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Ben Herfurth Priority: Major Blocker Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Ben Herfurth commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Latest LTS version 2.150.1 also has it. This really really annoying. We have to restart the complete server few times a day. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Mykola Marzhan commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" downgrade to jenkins-2.140 helped me, I don't see any issues anymore. btw, the issue exists on 2.138.3 and on 2.151 versions. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Vivek Pandey updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Vivek Pandey Labels: pipeline threads triaged-2018-11 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" After a longer period of not having experienced the problem, we just encountered it again. I could take a jstack snapshot and I found the following interesting thread stack trace: "Running CpsFlowExecution[Owner[apppname1/appname2/appname3/appname4/134675:appname1/appname2/appname3/appname4 #134675]]" #164285 daemon prio=5 os_prio=0 tid=0x7f668d046000 nid=0x2b16 waiting on condition [0x7f65db018000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd6edcc30> (a org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at org.codehaus.groovy.util.LockableObject.lock(LockableObject.java:37) at org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment.removeEntry(AbstractConcurrentMapBase.java:173) at org.codehaus.groovy.util.ManagedConcurrentMap$Entry.finalizeReference(ManagedConcurrentMap.java:81) at org.codehaus.groovy.util.ManagedConcurrentMap$EntryWithValue.finalizeReference(ManagedConcurrentMap.java:115) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.finalizeReference(GroovyClassValuePreJava7.java:51) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries0(ReferenceManager.java:108) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries(ReferenceManager.java:93) at org.codehaus.groovy.util.ReferenceManager$CallBackedManager.afterReferenceCreation(ReferenceManager.java:117) at org.codehaus.groovy.util.ReferenceManager$1.afterReferenceCreation(ReferenceManager.java:135) at org.codehaus.groovy.util.ManagedReference.(ManagedReference.java:36) at org.codehaus.groovy.util.ManagedReference.(ManagedReference.java:40) at org.codehaus.groovy.util.ManagedLinkedList$Element.(ManagedLinkedList.java:40) at org.codehaus.groovy.util.ManagedLinkedList.add(ManagedLinkedList.java:102) at org.codehaus.groovy.reflection.ClassInfo$GlobalClassSet.add(ClassInfo.java:478) - locked <0xd6e6aa68> (a org.codehaus.groovy.util.ManagedLinkedList) at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:83) at org.codehaus.groovy.reflection.ClassInfo$1.computeValue(ClassInfo.java:79) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue.(GroovyClassValuePreJava7.java:37) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:64) at org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment.createEntry(GroovyClassValuePreJava7.java:55) at org.codehaus.groovy.util.AbstractConcurrentMap$Segment.put(AbstractConcurrentMap.java:157) at
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Short update from our case: Yesterday we had another case where we almost stumbled into this situation again. Situation was that one user had caused many requests to the main page of a job (called "JenkinsJob102" later on) where the complex graphs (jacoco/test coverage/...) are rendered. We again stumbled into the situation that the HTTP request threads were hanging for a long time. CPU load was up to 500% (i.e. 5 cores were busy). Note that still some CPU capacity was free. I/O load apparently was not the biggest problem (otherwise we would have seen a different setting on "top"). Jenkins stopped job processing. HTTP response time was in the area of 30s and longer. We did not see the error log message "INFO: Running CpsFlowExecutionOwner[...] unresponsive for 5 sec" (or similar) yet, but the execution of Pipeline jobs seized. So I would have expected that this was to happen very soon. I attached a thread dump for you to this ticket (20181012-statebefore.txt). We detected two culprits: Lock 0xd7102070 was the culprit that the GET requests started to queue again (search for "#3231" in the thread dump file). All hanging GET HTTP requests threads were against JenkinsJob102. We first killed the thread which has "#3231" in its name, as it was the current owner of the lock. CPU shortly dropped, but all the rest of the other threads kicked in. We then manually killed also the rest of the threads, as we were very confident that these requests were leftovers which no user ever would require anymore. That took roughly 15 minutes to be done, as performance of Jenkins was bad. Once they were gone, CPU load was at around our usual 20%. Yet, the job queue was not processing anymore. Taking another thread dump snapshot (which unfortunately I lost shortly thereafter), we then detected that Yet-Another-Dockerplugin (YAD) was waiting for a response from our docker server again. It had a lock on the method "getClient()" and thus other threads for provisioning new slaves could not gain the lock (nearly all our jobs in the queue require a docker-based slave in one or the other way). Having cross-checked with the docker server (which was not expecting to send anything anymore), we then also killed that thread which was waiting for the answer from the docker server, which would never come. With that also provisioning of slaves resumed and the job queue started to reduce. Having the Jenkins server in good shape again, we dared to try reproducing the situation: One user logged on and opened four browser tabs pointing to the main page of job "JenkinsJob102". He then did a single "browser refresh (F5)" for each of these tabs. CPU load then almost immediately was up to 500% again and we had roughly a dozen of hanging GET request threads (note though that the number of "hanging threads" was much lower than I
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Nico Schmoigl Attachment: 20181012-statebefore.txt Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title John Wu commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" We encountered the same issue (Running Jenkins ver. 2.60.3). I also attached out thread-dump fyi. Restarting the master node solved the issue (for now). Our Jenkins is deployed to Kubernetes but we don't have Yet Another Docker plugin installed. jenkins-thread-dump.txt Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title John Wu updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: John Wu Attachment: jenkins-thread-dump.txt Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title John Wu updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: John Wu Attachment: jenkins-thread.dump.xml Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title John Wu updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: John Wu Attachment: jenkins-thread.dump.xml Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" We did some improvements over the last few releases of workflow-cps and workflow-job That sounds promising. We'll have a look at this some time later. Now, what I'd love to see is the full stack trace for that "endless loop". Before going into detail here, please let me reiterate one thing: It is not proven to me that the "endless loop" case really was an "early version" of the original bug report here. It has happened on the same server at roughly the same time. Having said this, let's have a look at the thread trace. I have attached an anonymized version of the thread trace I had created when the system was in that "endless loop" state (20180919-hangingjenkinsthreads-logs.txt). I suggest to start your analysis with searching for the term "main-voter", which is one of our jobs - and based on my analysis is the job, which caused the situation. Although we enabled quite strict retention on that job, we still have ~250 builds with it. Moreover, expect that each (successful) build will have around 1600 (mostly very small) log files in the build's folder (BTW they give us also a hard time with our backup strategy). Also, what is "YAD plugin" short for? "Yet Another Docker plugin"? Yes, correct. If so, I'm very curious how that could be related because the timeout there applies to communications with the Docker server. Well, this is yet another guess of us - here's the story: Remember that I had written Afterwards, we had to restart the server The reason for that was the thread "jenkins.util.Timer 6". If you look at its stack, you'll see that it's blocked in a ListContainersCmdExec request. That's a call via HTTP REST to the docker server, asking for the list of all containers running on the host (mainly - it's a little more complicated than this ). With an additional tool, we found out that it must be hanging there for hours (so much on "read timeout" - empty setting there means "infinity"). It's in blocked I/O state, waiting for the result coming back from the docker host. We don't exactly know what the docker host did (replied?), but usually such calls only take 1-2 seconds to answer - on very busy hosts it may be up to half a minute or so. You may expect that our docker host should respond within less of a second. Apparently, the missing response had blocked the YAD plugin and no further containers could be created (which mainly meant that the build queue was blocked, as nearly all our jobs require a new container). We could observe that this also had a negative effect on the management of already running containers/nodes for currently-running jobs (containers were hanging strangely). It wouldn't surprise me, if that also had indirect bad
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Nico Schmoigl Attachment: 20180919-hangingjenkinsthreads-logs.txt Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Sam Van Oort commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Nico Schmoigl I'm glad to see you haven't had a recurrence. That sounds like you were held up due to I/O issues – only one operation can try to load a build from disk at a time, and if that takes a while and other operations are depending on that build being loaded they will be blocked until it finishes loading. We did some improvements over the last few releases of workflow-cps and workflow-job to support lazy-loading of the FlowExecution which can make these cases significantly faster (especially where we are using the Performance-Optimized durability mode). Looking at performance-plugin, it seems that the Performance plugin causes this due to trying to load all builds for each run (which is bound to create a bottleneck for other things trying to access the builds as well). Now, what I'd love to see is the full stack trace for that "endless loop" – it's possible in some rare cases if the build onLoad ends up invoking itself either directly or indirectly, but suggests a critical bug somewhere (perhaps Performance Plugin, perhaps a combination of plugins). Also, what is "YAD plugin" short for? "Yet Another Docker plugin"? If so, I'm very curious how that could be related because the timeout there applies to communications with the Docker server. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Intermediate feedback: So far, the problem did not reappear for our server. The only thing which we have changed (after the issue I had documented in https://issues.jenkins-ci.org/browse/JENKINS-52362?focusedCommentId=349554=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-349554 ) was the "read timeout" setting in the YAD configuration section from "empty" to 120. I will update this ticket in case we experience yet another situation where the server stops responding. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title
Nico Schmoigl edited a comment on JENKINS-52362
Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Just for the record: At the same server, we experienced a deadlock situation today, which _could_ _m__ay_ be related to this issue:Today, we had a very sluggish server (long latency in response). Checking, we found several hanging inbound GET requests which took ages (>1,100,000 ms) to complete. A thread dump showed that several threads were blocked by a lock in jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369), which was indirectly triggered by hudson.plugins.performance.actions.PerformanceProjectAction.doRespondingTimeGraph. Note that there was not just the performance plugin, but we also saw other GET requests, such as /job/.../test/trend (there also locked in jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber) or /job/.../jacoco/graph (also in jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber) were affected. After a little of analysis, we found one of the /job/.../performance/throughputGraph jobs running (state "running"), which apparently was in an endless loop. It also held the lock of the critical monitor, which blocked all the other requests. The interesting (triggering) block within this thread to me was: {noformat}...at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)at hudson.util.XStream2.unmarshal(XStream2.java:114)at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)at hudson.XmlFile.unmarshal(XmlFile.java:160)at org.jenkinsci.plugins.workflow.job.WorkflowRun.reload(WorkflowRun.java:603)at hudson.model.Run.(Run.java:325)at org.jenkinsci.plugins.workflow.job.WorkflowRun.(WorkflowRun.java:209)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:165)at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:142){noformat}Killing the thread did the trick - and the rest started to work again. Afterwards, we had to restart the server - but that was due to another problem, which is unrelated to this one here.However, the situation today was different than before: Today, we had a significant load average / CPU load during that situation. In the previous situation, load average / CPU load was very normal - also for hours before the blocking event. Given that the symptoms are different, I am currently not sure whether we just saw the "early stage" of yet-another occurrence of this issue, which we could cure with a courageous thread kill, or whether this was something totally different. For sure, it makes sense to closely look at the list of locks pending, if the issue reappears.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Just for the record: At the same server, we experienced a deadlock situation today, which could be related to this issue: Today, we had a very sluggish server (long latency in response). Checking, we found several hanging inbound GET requests which took ages (>1,100,000 ms) to complete. A thread dump showed that several threads were blocked by a lock in jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369), which was indirectly triggered by hudson.plugins.performance.actions.PerformanceProjectAction.doRespondingTimeGraph. Note that there was not just the performance plugin, but we also saw other GET requests, such as /job/.../test/trend (there also locked in jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber) or /job/.../jacoco/graph (also in jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber) were affected. After a little of analysis, we found one of the /job/.../performance/throughputGraph jobs running (state "running"), which apparently was in an endless loop. It also held the lock of the critical monitor, which blocked all the other requests. The interesting (triggering) block within this thread to me was: ... at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:114) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at hudson.XmlFile.unmarshal(XmlFile.java:160) at org.jenkinsci.plugins.workflow.job.WorkflowRun.reload(WorkflowRun.java:603) at hudson.model.Run.(Run.java:325) at org.jenkinsci.plugins.workflow.job.WorkflowRun.(WorkflowRun.java:209) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:165) at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:142) Killing the thread did the trick - and the rest started to work again. Afterwards, we had to restart the server - but that was due to another problem, which is unrelated to this one here. However, the situation today was different than before: Today, we had a significant load average / CPU load during that situation. In the previous situation, load average / CPU load was very normal - also for hours before the blocking event. Given that the symptoms are different, I am currently not sure whether we just saw the "early stage" of yet-another occurrence of this issue, which we could cure with a courageous thread kill, or whether this was something totally different. For sure, it makes sense to closely look at the list of locks pending, if the issue reappears.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Sam Van Oort commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Nico Schmoigl Ack, we'll wait untuil you have a chance to grab the thread dump – otherwise it'll depend on if this gets fixed by another bugfix. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Please could you attach a thread dump from Jenkins? ... will try to when the server goes down next time. It might become a little tricky, as this is a productive instance and if it is down, pressure is often high to get it back up running again as soon as possible. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Josiah Niedrauer commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Sam Van Oort I have not been able to reproduce this after weeks of heavy use. It may have been adding swap that fixed it, or it may have been that a new version was pushed to the lts tag. I am not sure. In either case, it is unlikely that I will be able to get you a thread dump. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Sam Van Oort commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Nico Schmoigl Please could you attach a thread dump from Jenkins? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Similar issue also reproducible here on an older machine. Setup (amongst others): Jenkins core 2.60.3 Yet Another Docker Plugin 0.1.0-rc47 Jenkins running in docker container, connecting to another docker server for running jobs on the slave. System locks up, but apparently continues running internally. The HTTP server can't be down entirely, as sending a GET to the endpoint /api/json (which we use for "availability pinging") kept responding at usual response times. Jenkins runs jobs, which are executed every 5 minutes, so we can track down the point in time of it quite well. I could cross-check: There was more than 120MB of free heap memory for the java process + further 3GB of RAM. Operating system and docker logs around that time seem very unsuspicious. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl assigned an issue to Unassigned Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Nico Schmoigl Assignee: Nico Schmoigl Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Nico Schmoigl assigned an issue to Nico Schmoigl Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Nico Schmoigl Assignee: Nico Schmoigl Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Sam Van Oort updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Sam Van Oort Labels: pipeline threads Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Sam Van Oort updated an issue Jenkins / JENKINS-52362 Jenkins hangs due to "Running CpsFlowExecution unresponsive" Change By: Sam Van Oort Component/s: durable-task-plugin Component/s: workflow-cps-plugin Component/s: workflow-durable-task-step-plugin Component/s: core Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Sam Van Oort commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Josiah Niedrauer Philip Douglas Please can you grab and attach a thread dump from when you see this issue? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Josiah Niedrauer commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" I have not been able to reproduce this since adding swap to my jenkins master docker host. I think this condition may somehow be triggered by low memory. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52362) Jenkins hangs due to "Running CpsFlowExecution unresponsive"
Title: Message Title Josiah Niedrauer commented on JENKINS-52362 Re: Jenkins hangs due to "Running CpsFlowExecution unresponsive" Also seeing this issue. It starts looping with the unresponsive time going up, but otherwise no change. It never seems to recover from this state, even after 12+ hours. Running jenkins from the Docker tag: jenkins/jenkins:lts Currently at version: 2.121.3 Log output: Sep 01, 2018 11:42:41 PM com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform getSelectedProtocol INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? Sep 01, 2018 11:43:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0 INFO: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#570]: checking REDACTED on Docker (i-06939e1a358dc4ce5) unresponsive for 5 sec Sep 01, 2018 11:43:27 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0 INFO: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#570]: checking REDACTED on Docker (i-06939e1a358dc4ce5) unresponsive for 13 sec Sep 01, 2018 11:44:00 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
