[JIRA] (JENKINS-51241) Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2.
Title: Message Title Simon Watts created an issue Jenkins / JENKINS-51241 Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2. Issue Type: Improvement Assignee: Unassigned Attachments: jenkins-jheap.txt, jenkins-jinfo.txt, jenkins-jstack.txt Components: core Created: 2018-05-10 09:39 Environment: Jenkins 2.107.2 Red Hat Enterprise Linux Server release 6.6 (Santiago) x86_64 java version "1.8.0_51" Java(TM) SE Runtime Environment (build 1.8.0_51-b31) Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode) Priority: Critical Reporter: Simon Watts Since upgrading our Jenkins master from 2.89.2 to 2.107.2 (LTS) we have experienced the web interface becoming non-responsive (including jenkins-cli). Restarting the service clears the issue, but it keeps reappearing. SCM triggered builds appear to be continuing in the background. We did not experience this with the previous release. The stack trace reports a deadlock. Attached are the output from jinfo, jmap (heap), and jstack: jstack -l jmap -heap jinfo
[JIRA] (JENKINS-51241) Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2.
Title: Message Title Simon Watts edited a comment on JENKINS-51241 Re: Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2. We just restarted the Jenkins service. I managed to get into the web front end briefly, but it then hung again (so about 10 minutes since restart, and only a couple of minutes on the gui).Possible that there is an on-going job which regenerates the deadlock when the service restarts? Is there a way to stop jobs without access to the web or REST? I would like to abort the jobs listed in connection with the deadlock.Also to note: Plugins were all up-to-date as of *4 May 2018* at 10:40 hrs (UTC+1) .I note that there have been updates to pipeline plugins since that time, but if I cannot access the GUI or REST it is going to be hard to update them... Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51241) Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2.
Title: Message Title Simon Watts edited a comment on JENKINS-51241 Re: Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2. We just restarted the Jenkins service. I managed to get into the web front end briefly, but it then hung again (so about 10 minutes since restart, and only a couple of minutes on the gui).Possible that there is an on-going job which regenerates the deadlock when the service restarts? Is there a way to stop jobs without access to the web or REST? I would like to abort the jobs listed in connection with the deadlock. Also to note: Plugins were all up-to-date as of *4 May 2018* at 10:40 hrs.I note that there have been updates to pipeline plugins since that time, but if I cannot access the GUI or REST it is going to be hard to update them... Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51241) Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2.
Title: Message Title Simon Watts commented on JENKINS-51241 Re: Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2. We just restarted the Jenkins service. I managed to get into the web front end briefly, but it then hung again (so about 10 minutes since restart, and only a couple of minutes on the gui). Possible that there is an on-going job which regenerates the deadlock when the service restarts? Is there a way to stop jobs without access to the web or REST? I would like to abort the jobs listed in connection with the deadlock. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51241) Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2.
Title: Message Title Simon Watts commented on JENKINS-51241 Re: Jenkins web interface - deadlock since u/g to 2.107.2 from 2.89.2. Sam Van Oort thanks - back up and running Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49525) Rare exception sync'ing global pipeline library with new project
Title: Message Title Simon Watts commented on JENKINS-49525 Re: Rare exception sync'ing global pipeline library with new project We are now seeing this quite a bit when something triggers a lot of projects to rebuild. Currently running on the latest plugins and Jenkins 2.107.2. Same exception as given in the description. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49525) Rare exception sync'ing global pipeline library with new project
Title: Message Title Simon Watts created an issue Jenkins / JENKINS-49525 Rare exception sync'ing global pipeline library with new project Issue Type: Bug Assignee: Unassigned Components: p4-plugin Created: 2018-02-13 11:45 Environment: Jenkins 2.89.2 P4 Plugin 1.8.4 Pipeline 2.5 Priority: Minor Reporter: Simon Watts I've seen this a couple of times now – when first building a new project in Jenkins, an exception is thrown when loading the global pipeline library used by all our projects. This does not happen every time. There seems to be some confusion with the exception mentioning a job folder for a different project (see below). The exception has not been seen to reoccur when another build is triggered immediately after this failure (this may be due to the low frequency of this occurance). We have not seen this type of failure in projects which are not new – so seems related to the initial checkout? Note that no other projects where building at the time, and none where scheduled to build. The 'ANOTHERPROJECT' in the logs below was not recently rebuilt, and is not lexcially close to the project being built. Complete log for the initial failed build (with named redacted): Started by user anonymous Obtained DEPOT/PROJECT/Jenkinsfile from p4-HASH-PROJECT Running in Durability level: MAX_SURVIVABILITY Loading library LIBRARY@1.0 (p4):cmd:... p4 client -o extlib-LIBRARY p4 client -o extlib-LIBRARY (p4):stop:4 (p4):cmd:... p4 info p4 info (p4):stop:5 (p4):cmd:... p4 client -o extlib-LIBRARY p4 client -o extlib-LIBRARY (p4):stop:6 (p4):cmd:... p4 client -i p4 client -i Client
[JIRA] (JENKINS-49393) Broken symlinks in reused workspaces
Title: Message Title Simon Watts commented on JENKINS-49393 Re: Broken symlinks in reused workspaces Note that some symlinks are cleaned out, while others are not. It is possible that this is related to whether the symlink or target is processed first. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49628) Perforce "Workspace Name Format" reset to literal
Title: Message Title Simon Watts updated an issue Jenkins / JENKINS-49628 Perforce "Workspace Name Format" reset to literal Change By: Simon Watts Attachment: JENKINS-49628-1.png Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49629) Provide Perforce environment for scripts
Title: Message Title
Simon Watts created an issue
Jenkins / JENKINS-49629
Provide Perforce environment for scripts
Issue Type:
Improvement
Assignee:
Unassigned
Components:
p4-plugin
Created:
2018-02-19 15:04
Environment:
Jenkins 2.89.2 P4 Plugin 1.8.5
Priority:
Minor
Reporter:
Simon Watts
Our Jenkins build wraps build scripts and makefiles which are also used by developers. Some scripts will use the Perforce p4 command during their invocation – for example, to determine the mapped path of a file or directory, or to extract the version number of a file for documentation purposes. Developers already have their environment set-up and will be using p4 both on the command line, from IDEs, and from within the build scripts. However the Jenkins build does not have a valid Perforce environment as required by p4, even though it the build is operating with a workspace. Please make the P4 environment that defines the workspace available to shell commands invoked from the Jenkinsfile. If necessary, via a "withEnv()" type context. Workaround I acknowledge that the P4 Plugin exports a set of variables to the environment; however these do not have the naming required by p4. Therefore I have a custom step which uses these to create P4 Config and P4 Tickets files in the root of the workspace, and exports a P4CONFIG variable. This needs to be called in each stage of the pipeline in which shell commands may invoke p4. Have I overlooked something? Is there a {{"withEnv ("p4") {...}"-}}like step already that I don't know about that will do this?
[JIRA] (JENKINS-49628) Perforce "Workspace Name Format" reset to literal
Title: Message Title
Simon Watts created an issue
Jenkins / JENKINS-49628
Perforce "Workspace Name Format" reset to literal
Issue Type:
Bug
Assignee:
Unassigned
Components:
p4-plugin
Created:
2018-02-19 14:46
Environment:
Jenkins 2.89.2 P4 Plugin 1.8.5 Java: 1.8.0_51-b31 (Oracle, 64-bit) Master on RedHat Enterprise Linux 6.6 x86_64. Perforce Server version: P4D/LINUX26X86_64/2017.1/1545029 (2017/08/16)
Priority:
Major
Reporter:
Simon Watts
When configuring a Pipeline project with source from Perforce, it is required to set a "Workspace Name Format" as a template which can be used to generate per-node, per-executor workspace names required by Perforce. The default value for this field is adequate and we have not felt the need to change it:
jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}
Furthermore the field requires that the value contains at least one substitution variable to be accepted. I recently noticed that our project's workspace name formats had changed to literal values such as:
jenkinsTemp-87602970-2871-43c1-9759-0b16964ffa42
Which does not match any expansion of the original name format. This change also corresponds to issues recently observed whereby
[JIRA] (JENKINS-49628) Perforce "Workspace Name Format" reset to literal
Title: Message Title
Simon Watts updated an issue
Jenkins / JENKINS-49628
Perforce "Workspace Name Format" reset to literal
Change By:
Simon Watts
When configuring a Pipeline project with source from Perforce, it is required to set a _*"Workspace Name Format"*_ as a template which can be used to generate per-node, per-executor workspace names required by Perforce.The default value for this field is adequate and we have not felt the need to change it:{noformat}jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}{noformat}Furthermore the field requires that the value contains at least one substitution variable to be accepted.I recently noticed that our project's workspace name formats had changed to literal values such as:{noformat}jenkinsTemp-87602970-2871-43c1-9759-0b16964ffa42{noformat}Which does not match any expansion of the original name format.This change also corresponds to issues recently observed whereby developers have stopped getting emails on failed builds (which was previously working). In fact, even though builds are triggered by SCM changes, the associated change list is empty.We presume that this is a consequence of the Jenkins build now using the same workspace name for different build nodes and executors – I believe that Perforce frowns on such things.Having changed all projects back to the default template on Friday evening, looking again today and every project which has been rebuilt since then once again has a literal value in place of the template.*Light weight checkout?*_Since this wasn't happening before, we look for something that has changed..._The most likely candidate is that all projects were recently switched to using _"Lightweight Checkout"_ to obtain the Jenkinsfile – this capability became supported by P4 Plugin with release 1.8.4. This mechanism saves a lot (in our case) of disk space by avoiding the need for an additional workspace sync just to get the Jenkinsfile.This would also be in line with the literal workspace name appearing in project configurations not being related to the original template format.*Hypothesis*P4 Plugin light weight checkout of the Jenkins file is overwriting the workspace name template for projects with the literal workspace name used during the lightweight checkout, and rendering the project incompatible with Perforce.*Scenario*Our builds are principle C/C++ code built using a makefile, driven by a script which sets the variables as appropriate to the build host and target.Each project interacts with Perforce as follows: # A template workspace defined in Perforce. # A Jenkinsfile from Perforce in the project's home directory – using lightweight checkout. # A Jenkins extension library also held in Perforce is loaded by default. # The project sources from Perforce as defined by the template workspace.This has been the same for a while now, and the recent change of enabling lightweight checkout on the Jenkinsfile (2) seems the most likely candidate for triggering the bug. *Attached:* * Pipeline configuration with corrupted
[JIRA] (JENKINS-49628) Perforce "Workspace Name Format" reset to literal
Title: Message Title Simon Watts commented on JENKINS-49628 Re: Perforce "Workspace Name Format" reset to literal Confirmed that enabling "lightweight checkout" is the cause: Project has correct template for "Workspace Name Format" and "lightweight checkout" is enabled. Build the Project. Project now has the corrupted literal in the "Workspace Name Format" field. Correct "Workspace Name Format" field and disabled "lightweight checkout" of the Jenkins file. Build the Project. Project still has the correct template value in its "Workspace Name Format" field. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49678) Pipleine lightweight Jenkinsfile checkout still runs p4 sync after 'p4 print' of Jenkins file
Title: Message Title Simon Watts commented on JENKINS-49678 Re: Pipleine lightweight Jenkinsfile checkout still runs p4 sync after 'p4 print' of Jenkins file Has this issue actually been fixed, or was it delegated to JENKINS-49714 which was closed as a "won't fix"? Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49714) Set skipDefaultCheckout to false when using lightweight checkout for Jenkinsfile
Title: Message Title Simon Watts commented on JENKINS-49714 Re: Set skipDefaultCheckout to false when using lightweight checkout for Jenkinsfile Please see JENKINS-49678 for the reason for this JIRA being raised. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49525) Rare exception sync'ing global pipeline library with new project
Title: Message Title Simon Watts updated an issue Jenkins / JENKINS-49525 Rare exception sync'ing global pipeline library with new project Change By: Simon Watts Attachment: JENKINS-49525-1.txt Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49525) Rare exception sync'ing global pipeline library with new project
Title: Message Title Simon Watts commented on JENKINS-49525 Re: Rare exception sync'ing global pipeline library with new project Just encountered this behaviour in an established project from an overnight build – the console output is attached as JENKINS-49525-1.txt (with names redacted). In this case there is evidence that 'ANOTHERPROJECT' was built successfully at around the same time and the 'PROJECT' which failed. These projects are using template workspaces defined in Perforce, and are not using light weight checkout. Other projects where also built at around the same time (successfully or failing due to code errors). Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
