[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort started work on JENKINS-49686 Change By: Sam Van Oort Status: Reopened In Progress Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort closed an issue as Done Should be resolved with release of workflow-cps 2.50 and workflow-job 2.21 CC bentoi Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Status: In Progress Closed Resolution: Done Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more opt
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead bentoi Appreciate the update – I think I've got a handle on all of the related issues here and test coverage for it. Just working through a few remaining challenges and then review + human testing. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Jenn Briden updated an issue Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Jenn Briden Sprint: Pipeline - April 2018 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title bentoi commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead I no longer see the NullPointerException with the latest plugins however I'm seeing some failures when trying to load the "FlowNodes" Apr 25, 2018 2:21:04 AM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution initializeStorage WARNING: Tried to load head FlowNodes for execution Owner[ice/3.7/264:ice/3.7 #264] but FlowNode was not found in storage for head id:FlowNodeId 1:1858 Apr 25, 2018 2:21:04 AM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution rebuildEmptyGraph WARNING: Failed to load pipeline heads, so faking some up for execution CpsFlowExecution[Owner[ice/3.7/264:ice/3.7 #264]] Apr 25, 2018 2:21:04 AM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoad WARNING: Completed flow without FlowEndNode: CpsFlowExecution[Owner[ice/3.7/264:ice/3.7 #264]] heads:1859::1859:org.jenkinsci.plugins.workflow.graph.FlowStartNode[id=1860] These failures are also mentioned on JENKINS-50199. Let me know if you need any additional information related to these. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+u
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort reopened an issue At least one report still in JENKINS-50199 Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Resolution: Fixed Status: Closed Reopened Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort updated JENKINS-49686 Resolved as of workflow-cps 2.47 and workflow-job 2.18 Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Status: In Review Closed Resolution: Fixed Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.googl
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort updated an issue Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Component/s: workflow-job-plugin Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title SCM/JIRA link daemon commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Code changed in jenkins User: Sam Van Oort Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunRestartTest.java src/test/java/org/jenkinsci/plugins/workflow/job/WorkflowRunTest.java http://jenkins-ci.org/commit/workflow-job-plugin/f0c26058f31d4f159a82a3cace52935e93f20701 Log: Merge pull request #93 from svanoort/fix-resume-issues Fix resume issues JENKINS-49686 and JENKINS-50199 and JENKINS-50407 Compare: https://github.com/jenkinsci/workflow-job-plugin/compare/e11cea623f61...f0c26058f31d Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead bentoi I've got some proposed fixes that should resolve... well the issue here plus a whole beeshive of related bugs with similar causes. Plus it adds some catchall protections that protect against unseen but possible issues. Please could you try out the below plugins (snapshot builds of PRs in final review) and confirm it resolves issues fully for you: workflow-cps.hpi workflow-job.hpi Or if you prefer to build them yourself, the PRs are here: https://github.com/jenkinsci/workflow-cps-plugin/pull/216 https://github.com/jenkinsci/workflow-job-plugin/pull/93 Thanks! Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort updated an issue Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Attachment: workflow-job.hpi Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort updated an issue Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Attachment: workflow-cps.hpi Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort updated JENKINS-49686 Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Status: In Progress Review Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead On second thought... no, not quite, but that feels very thread-unsafe. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Jesse Glick Won't that open up other race conditions? Adding an Action to the flowNode is safe, but this can be invoking "finish" on the Run. Perhaps the listener should block until completed is non-null? Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title
Jesse Glick edited a comment on JENKINS-49686
Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Just realized this was on my list of “Pipeline test flakes I have not gotten around to filing/fixing yet”:{code:none}java.lang.NullPointerException at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:959) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1220) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:408) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) at …{code}in{code:java}synchronized (completed) {{code}Seen randomly during a in some test in SECURITY-699 {{org.jenkinsci.plugins.workflow.support.pickles.serialization.SerializationSecurityTest}} which breaks resumption of a build.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title
Jesse Glick commented on JENKINS-49686
Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Just realized this was on my list of “Pipeline test flakes I have not gotten around to filing/fixing yet”:
java.lang.NullPointerException
at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:959)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1220)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:408)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
at …
in
synchronized (completed) {
Seen randomly during a test in https://github.com/jenkinsci-cert/workflow-support-plugin/pull/1 which breaks resumption of a build.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jen
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title
Jesse Glick edited a comment on JENKINS-49686
Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Just realized this was on my list of “Pipeline test flakes I have not gotten around to filing/fixing yet”: {code:none}java.lang.NullPointerException at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:959) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1220) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:408) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) at …{code} in {code:java}synchronized (completed) {{code} Seen randomly during a test in https://github.com/jenkinsci SECURITY - cert/workflow-support-plugin/pull/1 699 which breaks resumption of a build.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Jesse Glick commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Just drop the synchronization in GraphL if completed == null; in this case there is no need for synchronization anyway. Also this code is slated for deletion in JENKINS-38381. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Got it – and it's a DOOZY of a race condition that I'm lucky enough to be able to reproduce for the moment: When loading the pipeline build, onLoad is called on the WorkflowRun, which then calls onLoad on the FlowExecution if non-null: https://github.com/jenkinsci/workflow-job-plugin/blob/357b944fd8651c8d2a0f238cfbee5185e1c0eef3/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java#L635 This is invoked before initializing the "completed" boolean in the Run. Unfortunately as part of the cleanup if the execution is in an invalid state, perhaps due to incomplete persistence, the execution will create new FlowHeads, by calling FlowHead.onNewHead, which will notify listeners: https://github.com/jenkinsci/workflow-cps-plugin/blob/b3d7ea363736d62c66c355514fa64c4eb8223434/src/main/java/org/jenkinsci/plugins/workflow/cps/FlowHead.java#L134 However, the WorkflowRun has one such listener, which will try to synchronize on the un-initialized 'completed' field if not set: https://github.com/jenkinsci/workflow-job-plugin/blob/357b944fd8651c8d2a0f238cfbee5185e1c0eef3/src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java#L995 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort started work on JENKINS-49686 Change By: Sam Van Oort Status: Open In Progress Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead So here's what I've got from digging (mostly to save my notes and provide transparency in case someone else sees something): The build nominally completed with a failure (tests failed, and you ran an Error step which was not caught) in FlowEndNode ID 1692 But somehow the build.xml shows: 1696 1693:1696 (flowHead id : flowNodeId for that head) And there are no flowNodes after 1692 in the storage. So what I'm trying to figure out is: How did the iota, head, and finalFlowNode ID get bumped above 1692? Does this occur with simpler builds? Or does it only happen when there's some specific structure including parallels (as with this one)? Why are we even trying to resume the build? We're done, kaput, finished, finito! The FlowEndNode has been written, and build ended with Failure result (as it should). This points to an issue specifically with the closedown of the build or checks on whether we need to resume the build (i.e. finished build showing as incomplete). The second part first: there's a 'done' flag on the CpsFlowExecution which if true marks the execution as complete. ODDLY this is absent in the persisted build record, and that appears to be normal somehow (going back to v2.0 of this plugin it is not persisted by the ConverterImpl that does marshall/unmarshall). Failing that, we look for having just a single head FlowNode (check!) and that head being a FlowEndNode (should have been the case?) Now for the end-of-build log: the execution sets itself to 'done' (which is not persisted, since that logic was not changed), sets the stored heads to just the first head (should be the FlowEndNode), and the flushes FlowNode storage. However, the WorkflowRun saves itself when done. Possible solutions: 1. Explicitly save 'done' in the flowExecution 2. Figure out why the FlowEndNode isn't ending up as the final head for the CpsFlowExecution thus signalling we're done done done. Add Comment
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title bentoi updated an issue Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: bentoi Attachment: build.xml.gz Attachment: flowNodeStore.xml.gz Attachment: jenkins.log.gz Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title bentoi commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Thanks for your response, here's my answers to your questions. We use: org.jenkins-ci.plugins.workflow:workflow-job:2.17 org.jenkins-ci.plugins.workflow:workflow-cps:2.45 We started using PERFORMANCE_OPTIMIZED recently to see if it helped working around JENKINS-49646. That's also when we started seeing this problem which now occurs quite frequently. We usually don't restart Jenkins while JOBs are running. We either use "systemctl restart jenkins" or a "plugin update + restart". This problem causes our job queue to hang frequently because some invalid jobs appear to consume executors. When this occurs we restart jenkins and it usually allows the pending job to start but then the problem eventually shows up again requiring to restart jenkins again. The NPE does indeed appear to occur after the restart. I've attached build.xml / flownodeStore.xml for the latest NPE from jenkins.log (build 233). Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title bentoi updated an issue Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: bentoi Attachment: jenkins.log.gz Attachment: flowNodeStore.xml.gz Attachment: build.xml.gz Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead This line indicates we did detect incomplete/inconsistent persistence of data – the build should have failed promptly and cleanly at this point: > WARNING: Pipeline state not properly persisted, cannot resume job/ice/job/3.7/221/ Somehow the CpsVmExecutorService is still continuing to run threads for the Pipeline even after it should have failed cleanly. Possible candidate: `FlowExecutionList.ItemListenerImpl.onLoaded()` Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Analysis to continue once we have more info: I've fixed a bunch of cases that result in similar issues – the root cause is a mismatches between the FlowNode storage and either the Pipeline Program or the Build.xml. This usually happens when the master restarts and for whatever reason is unable to load the FlowNode matching a FlowHead – this is why I ask about the restart and restart method, build.xml, and last Pipeline FlowNode from the storage. A "graceful shutdown" (https://jenkins.io/doc/book/pipeline/scaling-pipeline/#what-am-i-giving-up-with-this-durability-setting-trade-off) should force persistence of any un-saved data but it's possible for there to be subtle ordering issues if somehow execution continued after that (it shouldn't, but there could be bugs). I'm looking for the Jenkins Log to see if there were other issues associate with this – I'd expect this has generated other stack traces potentially, especially around restart. An error like 'List of Flow Heads Unset for...' would point the build.xml (containing the CpsFlowExecution info) being behind the program state (might need another persistence call somewhere). Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead bentoi This should be possible to solve, but I'm going to need a bit fair bit of info to diagnose this one, I'm afraid, because the cause is complex: 1. Which versions of Workflow-CPS and Workflow-Job plugins are you running? 2. What Durability Setting is your pipeline running in? (Should be at the top of your build log, something like MAX_SURVIVABILITY or PERFORMANCE_OPTIMIZED). 3. Did you recently restart the master while the pipelines were running, and if so, by which method did you restart it? (visiting the /restart url, service jenkins restart, pkill java, etc). 4. Please can you attach a copy of the Jenkins.log covering the time the build started until these errors appeared? 5. Please can you attach the build.xml for the build? 6. In the build's workflow directory, there's either a flownodeStore.xml file or a bunch of tiny XML files. Please, can you upload the entry with the highest ID (they'll be numbered 2, 3, 4, etc) – if it's the flownodeStore.xml it's fine to just GZIP it and attach it. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Sam Van Oort assigned an issue to Sam Van Oort Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Change By: Sam Van Oort Assignee: Sam Van Oort Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title Andrew Bayer commented on JENKINS-49686 Re: NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Hmm, I'm pretty sure this would be blocking the builds, yes. Not sure how we could end up with a null FlowNode or a null logsToCopy in GraphL#onNewHead, though. Sam Van Oort - any thoughts? Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49686) NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead
Title: Message Title bentoi created an issue Jenkins / JENKINS-49686 NPE in CPS VM thread at WorkflowRun$GraphL.onNewHead Issue Type: Bug Assignee: Unassigned Components: workflow-cps-plugin Created: 2018-02-22 07:36 Environment: Jenkins 2.89.4 Priority: Major Reporter: bentoi I have 2 jobs stuck in the build queue waiting, jobs are apparently waiting for 2 other jobs to complete but the nodes executors are free. I don't if these NPE can cause this behavior but they don't look right anyway. Feb 21, 2018 8:38:55 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution onLoad WARNING: Pipeline state not properly persisted, cannot resume job/ice/job/3.7/221/ Feb 21, 2018 8:38:55 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[ice/3.7/221:ice/3.7 #221]] java.lang.NullPointerException at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:997) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1368) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:412) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.
