[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Jesse Glick updated JENKINS-41945 Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Jesse Glick Status: In Review Resolved Resolution: Fixed Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title SCM/JIRA link daemon commented on JENKINS-41945 Re: Eventual NPE while running many concurrent declarative pipelines Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java http://jenkins-ci.org/commit/workflow-cps-plugin/1155b889193713f7fedaef8de8688128136d4fe4 Log: [FIXED JENKINS-41945] Ignore null elements in ClassInfo.globalClassSet.items. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Jesse Glick Component/s: pipeline-model-definition-plugin Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Jesse Glick updated JENKINS-41945 Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Jesse Glick Status: In Progress Review Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Jesse Glick started work on JENKINS-41945 Change By: Jesse Glick Status: Open In Progress Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Jesse Glick commented on JENKINS-41945 Re: Eventual NPE while running many concurrent declarative pipelines These standard pipelines feel like they run considerably faster than the declarative ones from before. Not surprising; I think to compare apples to apples you would need to make the standard Pipeline scripts much more complicated. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Karl Shultz commented on JENKINS-41945 Re: Eventual NPE while running many concurrent declarative pipelines Ran a test with 60 declarative pipelines running every minute, instead of 100. Still hit an out of memory condition. I'll attach a tarball of the logs. Inside that, jenkins.log.1 has a lot more historical data in it. jenkins.log is sort of its last gasp. Jenkins also generated an error report file. I'll attach that as well. logs.tar.gz hs_err_pid27051.log Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Karl Shultz updated an issue Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Karl Shultz Attachment: logs.tar.gz Attachment: hs_err_pid27051.log Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Karl Shultz commented on JENKINS-41945 Re: Eventual NPE while running many concurrent declarative pipelines The standard pipelines ran overnight, and are still running this morning. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Karl Shultz updated an issue Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Karl Shultz Attachment: screenshot-1.png Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Karl Shultz commented on JENKINS-41945 Re: Eventual NPE while running many concurrent declarative pipelines I'll let the standard ones run overnight, and see how things look tomorrow. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Karl Shultz commented on JENKINS-41945 Re: Eventual NPE while running many concurrent declarative pipelines jenkins--failure.log I let the standard pipelines run 20 minutes or so. There wasn't any sign of degraded performance to speak of. Here's the beginning and end of the vmstat output: procs ---memory-- ---swap-- -io -system-- --cpu- r b swpd free buff cache si sobibo in cs us sy id wa st 02/11/2017 01:47:41c 2 0 0 12885760 189508 8319160015 268 273 144 75 0 24 1 0 (snip) 02/11/2017 02:06:41c 0 0 0 12805028 191464 83247200 0 432 175 204 0 0 99 1 0 That amount of decrease in free memory is pretty small. The UI remains responsive throughout. To contrast, here's a similar timespan, with the declarative jobs running. Bu the time I stopped vmstat recording, the UI had stopped responding. procs ---memory-- ---swap-- -io -system-- --cpu- r b swpd free buff cache si sobibo in cs us sy id wa st 02/11/2017 02:25:33c 1 0 0 13255960 195416 9155000013 251 251 144 65 0 33 1 0 (snip) 02/11/2017 02:45:33c 4 0 0 11748496 199188 98192800 0 1471 1290 735 93 0 7 0 0 The declarative jobs are consuming more resource, but I think that's expected. The log, which I'll attach as jenkins--failure.log, is interesting. The jobs run for a while. Then I start to see these: Feb 11, 2017 2:27:34 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: DeclarativePipelines/inlineThing72 #80 completed: SUCCESS Feb 11, 2017 2:27:35 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: DeclarativePipelines/inlineThing70 #80 completed: SUCCESS Feb 11, 2017 2:27:39 AM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution cleanUpHeap WARNING: failed to clean up memory from Owner[DeclarativePipelines/inlineThing63/81:DeclarativePipelines/inlineThing63 #81] java.lang.NullPointerException at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57) at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38) at java.lang.reflect.Field.get(Field.java:393) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpGlobalClassSet(CpsFlowExecution.java:1034) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:966) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:968) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:968) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:957) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.cleanUpLoader(CpsFlowExecution.java:957) at
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Karl Shultz updated an issue Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Karl Shultz Attachment: jenkins--failure.log Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title
Karl Shultz commented on JENKINS-41945
Re: Eventual NPE while running many concurrent declarative pipelines
I ran the following standard pipeline script in the same way (100 concurrently, every minute):
node {
stage("TestingTesting") {
sh 'echo Hello world'
}
}
These standard pipelines feel like they run considerably faster than the declarative ones from before. So it could be that I'll need to run way more than 100 of them to reproduce the issue with standard pipeline. This time, I've got something running to tell me about system utilization:
vmstat 15 | awk '{system("X=$(date +\"%x %T\"); echo $X\\\c");print $0}' > vmstat.15.standard &
(Thanks Google!) I'll do likewise the next time I go to recreate this with declarative pipelines.
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Andrew Bayer updated an issue Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Andrew Bayer Environment: openjdk version "1.8.0_45-internal"Ubuntu 15.04 workflow-cps 2.26pipeline-model-definition 1.0.1core 2.32.2 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Andrew Bayer commented on JENKINS-41945 Re: Eventual NPE while running many concurrent declarative pipelines FYI - this line is the problem line in workflow-cps 2.26, which is what's installed. The NPE seems to be that classInfo is null. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Andrew Bayer assigned an issue to Jesse Glick Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Andrew Bayer Assignee: Andrew Bayer Jesse Glick Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title Andrew Bayer updated an issue Jenkins / JENKINS-41945 Eventual NPE while running many concurrent declarative pipelines Change By: Andrew Bayer Component/s: workflow-cps-plugin Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-41945) Eventual NPE while running many concurrent declarative pipelines
Title: Message Title
Karl Shultz created an issue
Jenkins / JENKINS-41945
Eventual NPE while running many concurrent declarative pipelines
Issue Type:
Bug
Assignee:
Andrew Bayer
Attachments:
jenkins.log
Components:
pipeline-model-definition-plugin
Created:
2017/Feb/10 10:11 PM
Environment:
openjdk version "1.8.0_45-internal" Ubuntu 15.04
Priority:
Major
Reporter:
Karl Shultz
After an hour or so of running 101 concurrent "hello world" pipeline scripts, the Jenkins server becomes unresponsive. The first signs of trouble in the log around here - I'll attach the entire log as well. The attached one probably has a bunch of noise in it from some earlier work. The pipeline script itself looks exactly like this:
pipeline {
agent any
stages {
stage("TestingTesting") {
steps {
sh 'echo Hello world'
}
}
}
}
INFO: DeclarativePipelines/inlineThing34 #32 completed: SUCCESS
Feb 10, 2017 8:27:44 PM org.jenkinsci.plugins.workflow.cps.CpsFlowExecution cleanUpHeap
WARNING: failed to clean up memory from
