| I have a pipeline Job that does some simple checks for all nodes every 5 minutes.
Sometimes (something like twice a day) this job fails with ConcurrentModificationException in ExwsAllocateActionImpl (on a Windows node while accessing a file on the network). This has been happening ever since but I didn't bother because the next build ran fine again. But since some updates (in July or August) the build hangs in a strange state after this exception.
Somehow the build is still running although it is failed already.
Therefore the next build doesn't run because it waits for the previous build to finish (I configured to not allow concurrent builds).
I can't abort the failed build because it is already failed.
So I have to delete the failed build in order to get the next build running. It doesn't happen anymore since I switched "Pipeline speed/durability override" from
performance optimized to maximum durability Log: Started by timer
Running in Durability level: PERFORMANCE_OPTIMIZED
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (master)
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] node
Running on Jenkins in /rsync/JenkinsJobs/workspace/work_ServerCheck
[Pipeline] {
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] isUnix
[Pipeline] sh
+ df /rsync
[Pipeline] sh
+ uptime
[Pipeline] isUnix
[Pipeline] isUnix
[Pipeline] echo
master: 37.99999952316284
[Pipeline] echo
we have 34710392 bytes left on master
[Pipeline] fileOperations
File Delete Operation:
/rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_master.txt deleting....
Success.
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] writeFile
[Pipeline] exwsAllocate
Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config
Using Disk allocation strategy: 'Select the Disk with the most usable space'
Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds'
The path on Disk is: unmerged/work_ServerCheck
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Running in /media/Builds/unmerged/work_ServerCheck
[Pipeline] { [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: /media/Builds/unmerged/work_ServerCheck/build_152473_master.txt deleting.... Success. [Pipeline] sleep Sleeping for 1 sec [Pipeline] } [Pipeline] // exws
[Pipeline] fileOperations
File Copy Operation:
/rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_master.txt
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Running in /media/Builds/unmerged/work_ServerCheck
[Pipeline] { [Pipeline] fileExists [Pipeline] }
[Pipeline] // exws
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (domvsaf1)
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] node
Running on domvsaf1 in /rsync/JenkinsJobs/workspace/work_ServerCheck
[Pipeline] { [Pipeline] sleep Sleeping for 1 sec [Pipeline] isUnix [Pipeline] sh + df /rsync [Pipeline] sh + uptime [Pipeline] isUnix [Pipeline] isUnix [Pipeline] echo domvsaf1: 0.9999999776482582 [Pipeline] echo we have 39575068 bytes left on domvsaf1 [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_domvsaf1.txt deleting.... Success. [Pipeline] writeFile [Pipeline] exwsAllocate Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config Using Disk allocation strategy: 'Select the Disk with the most usable space' Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds' The path on Disk is: unmerged/work_ServerCheck [Pipeline] exws Searching for disk definitions in the External Workspace Templates from Jenkins global config Running in /media/Builds/unmerged/work_ServerCheck [Pipeline] \{ [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /media/Builds/unmerged/work_ServerCheck/build_152473_domvsaf1.txt deleting.... Success. [Pipeline] }
[Pipeline] // exws
[Pipeline] fileOperations
File Copy Operation:
[Pipeline] sleep
Sleeping for 1 sec
/rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_domvsaf1.txt
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Running in /media/Builds/unmerged/work_ServerCheck
[Pipeline] {[Pipeline] fileExists[Pipeline] }
[Pipeline] // exws
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (SRV1625u)
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] node
Running on SRV1625u in /rsync/JenkinsJobs/workspace/work_ServerCheck
[Pipeline] {
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] isUnix
[Pipeline] sh
+ df /rsync
[Pipeline] sh
+ uptime
[Pipeline] isUnix
[Pipeline] isUnix
[Pipeline] echo
SRV1625u: 0.9999999776482582
[Pipeline] echo
we have 19422016 bytes left on SRV1625u
[Pipeline] fileOperations
File Delete Operation:
[Pipeline] sleep
Sleeping for 1 sec
/rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_SRV1625u.txt deleting....
Success.
[Pipeline] writeFile
[Pipeline] exwsAllocate
Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config
Using Disk allocation strategy: 'Select the Disk with the most usable space'
Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds'
The path on Disk is: unmerged/work_ServerCheck
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Running in /media/Builds/unmerged/work_ServerCheck
[Pipeline]
{ [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /media/Builds/unmerged/work_ServerCheck/build_152473_SRV1625u.txt deleting.... Success. [Pipeline] }
[Pipeline] // exws
[Pipeline] fileOperations
File Copy Operation:
[Pipeline] sleep
Sleeping for 1 sec
/rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_SRV1625u.txt
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Running in /media/Builds/unmerged/work_ServerCheck
[Pipeline]
{ [Pipeline] fileExists [Pipeline] } [Pipeline] // exws
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (SRV4708)
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] node
Running on SRV4708 in /rsync/JenkinsJobs/workspace/work_ServerCheck
[Pipeline] { [Pipeline] sleep Sleeping for 1 sec [Pipeline] isUnix [Pipeline] sh + df /rsync [Pipeline] sh + uptime [Pipeline] isUnix [Pipeline] isUnix [Pipeline] echo SRV4708: 0.0 [Pipeline] echo we have 363949504 bytes left on SRV4708 [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /rsync/JenkinsJobs/workspace/work_ServerCheck/build_152473_SRV4708.txt deleting.... Success. [Pipeline] writeFile [Pipeline] exwsAllocate Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config Using Disk allocation strategy: 'Select the Disk with the most usable space' Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds' The path on Disk is: unmerged/work_ServerCheck [Pipeline] exws Searching for disk definitions in the External Workspace Templates from Jenkins global config Running in /media/Builds/unmerged/work_ServerCheck [Pipeline] \{ [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec /media/Builds/unmerged/work_ServerCheck/build_152473_SRV4708.txt deleting.... Success. [Pipeline] } [Pipeline] // exws
[Pipeline] fileOperations
File Copy Operation:
[Pipeline] sleep
Sleeping for 1 sec
/rsync/JenkinsJobs/workspace/work_ServerCheck/build_152474_SRV4708.txt
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Running in /media/Builds/unmerged/work_ServerCheck
[Pipeline] {[Pipeline] fileExists[Pipeline] } [Pipeline] // exws
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (domvsafb1)
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] node
Running on domvsafb1 in e:\jenkins\workspace\work_ServerCheck
[Pipeline] {
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] isUnix
[Pipeline] sh
+ df /cygdrive/e/jenkins/workspace/
[Pipeline] sh
+ wmic cpu get loadpercentage
[Pipeline] isUnix
[Pipeline] isUnix
[Pipeline] echo
domvsafb1: 12.0
[Pipeline] echo
we have 12600340 bytes left on domvsafb1
[Pipeline] fileOperations
File Delete Operation:
[Pipeline] sleep
Sleeping for 1 sec
e:\jenkins\workspace\work_ServerCheck\build_152473_domvsafb1.txt deleting....
Success.
[Pipeline] writeFile
[Pipeline] exwsAllocate
Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config
Using Disk allocation strategy: 'Select the Disk with the most usable space'
Selected Disk ID 'S_Produkte__Builds' from the Disk Pool ID 'ArchivedBuilds'
The path on Disk is: unmerged/work_ServerCheck
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Searching for disk definitions in the Node config
Running in \\servername_Builds\unmerged\work_ServerCheck
[Pipeline] { [Pipeline] pwd [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec \\servername\_Builds\unmerged\work_ServerCheck\build_152473_domvsafb1.txt deleting.... Success. [Pipeline] } [Pipeline] // exws
[Pipeline] fileOperations
File Copy Operation:
[Pipeline] sleep
Sleeping for 1 sec
e:\jenkins\workspace\work_ServerCheck\build_152474_domvsafb1.txt
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Searching for disk definitions in the Node config
Running in \\servername_Builds\unmerged\work_ServerCheck
[Pipeline] { [Pipeline] fileExists [Pipeline] } [Pipeline] // exws
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (domvsafb2)
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] node
Running on domvsafb2 in e:\jenkins\workspace\work_ServerCheck
[Pipeline] { [Pipeline] sleep Sleeping for 1 sec [Pipeline] isUnix [Pipeline] sh + df /cygdrive/e/jenkins/workspace/ [Pipeline] sh + wmic cpu get loadpercentage [Pipeline] isUnix [Pipeline] isUnix [Pipeline] echo domvsafb2: 20.0 [Pipeline] echo we have 6471024 bytes left on domvsafb2 [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec e:\jenkins\workspace\work_ServerCheck\build_152473_domvsafb2.txt deleting.... Success. [Pipeline] writeFile [Pipeline] exwsAllocate Disk allocation strategy was not provided as step parameter. Fallback to the strategy defined in the Jenkins global config Using Disk allocation strategy: 'Select the Disk with the most usable space' [Pipeline] End of Pipeline java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) Caused: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.ewm.actions.ExwsAllocateActionImpl#allocatedWorkspaces for class org.jenkinsci.plugins.ewm.actions.ExwsAllocateActionImpl at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:263) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:250) Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:254) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:149) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.util.XStream2.toXMLUTF8(XStream2.java:313) at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34) at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1137) at hudson.BulkChange.commit(BulkChange.java:98) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1475) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Finished: FAILURE [Pipeline] exws Searching for disk definitions in the External Workspace Templates from Jenkins global config Searching for disk definitions in the Node config Running in \\servername_Builds\unmerged\work_ServerCheck [Pipeline] \{ [Pipeline] pwd (show) [Pipeline] fileOperations File Delete Operation: [Pipeline] sleep Sleeping for 1 sec \\servername\_Builds\unmerged\work_ServerCheck\build_152473_domvsafb2.txt deleting.... Success. [Pipeline] } [Pipeline] // exws
[Pipeline] fileOperations
File Copy Operation:
[Pipeline] sleep
Sleeping for 1 sec
e:\jenkins\workspace\work_ServerCheck\build_152474_domvsafb2.txt
[Pipeline] exws
Searching for disk definitions in the External Workspace Templates from Jenkins global config
Searching for disk definitions in the Node config
Running in \\servername_Builds\unmerged\work_ServerCheck
[Pipeline] {[Pipeline] fileExists[Pipeline] } [Pipeline] // exws
[Pipeline] sleep
Sleeping for 1 sec
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] node
Running on SRV4708 in /rsync/JenkinsJobs/workspace/work_ServerCheck
[Pipeline] { [Pipeline] fileOperations File Create Operation: [Pipeline] plot [Pipeline] fileOperations File Create Operation: [Pipeline] plot [Pipeline] archiveArtifacts Archiving artifacts [Pipeline] logParser [Pipeline] } [Pipeline] // node
data_load.csv file already exists, replacing the content with the provided content.
Creating file: /rsync/JenkinsJobs/workspace/work_ServerCheck/data_load.csv
data_disk.csv file already exists, replacing the content with the provided content.
Creating file: /rsync/JenkinsJobs/workspace/work_ServerCheck/data_disk.csv |
