Like I say, not my code as such, a Gen AI suggestion, but I would say it's an easy way for the process running as the jenkins user, to remove files which are not owned by the jenkins user, which seems to often be the reason for these ws-cleanup directories not being removed (Often in the Camel project case anyway). The other reason might be a restart of the server for example, which doesn't happen on lucene but sometimes does happen on some other CI servers we have due to Jenkins locking up (GC/RAM usage often).
On 2026/07/14 12:49:48 David Smiley wrote: > I modified one build, > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to > have an additional shell execution step with that snippet, which performs > the cleanup via Docker as you suggested, albeit not with the Jenkins/groovy > wrapper all around it. It ran successfully this morning. I'm not sure why > Docker is involved for what looks like a simple file removal; might you > explain? > > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <[email protected]> wrote: > > > Hi, > > > > In Infra, we don't work on job definitions as that is a project's domain. > > The above is only a suggestion which may or may not help. > > If it is easier, you could look at adding similar code to the jobs as a > > shell script post build step in order to remove the files. Aurélien Pupier > > of the Camel project has been looking at the same issue there, so worth > > having a chat with them about potential solutions. > > > > In terms of the jobs which are leaving ws-cleanup directories behind, it > > seems to be many of them. On lucene 3 today, there are already 113 such > > directories, 213GB so far. > > > > the jobs that they are associated with, from per their directory names are: > > > > OWASP-9.x > > OWASP-main > > Solr-Artifacts-10.x > > Solr-Artifacts-9.11 > > Solr-Artifacts-9.x > > Solr-Artifacts-main > > Solr-BadApples-Tests-main > > Solr-Docker-Nightly-9.11 > > Solr-Lint-10.x > > Solr-Lint-9.11 > > Solr-Lint-9.x > > Solr-Lint-main > > Solr-Maven-Snapshots-main > > Solr-Test-10.0 > > Solr-Test-10.x > > Solr-Test-9.11 > > Solr-Test-9.x > > Solr-Test-main > > Solr-TestIntegration-10.x > > Solr-TestIntegration-9.11 > > Solr-TestIntegration-9.x > > Solr-TestIntegration-main > > solr-desktop-ui-nightly > > > > On 4, there are 80 from these builds: > > > > OWASP-9.x > > OWASP-main > > Solr-Artifacts-10.x > > Solr-Artifacts-9.11 > > Solr-Artifacts-9.x > > Solr-Docker-Nightly-9.11 > > Solr-Lint-10.x > > Solr-Lint-9.11 > > Solr-Lint-9.x > > Solr-Lint-main > > Solr-Maven-Snapshots-main > > Solr-Test-10.0 > > Solr-Test-10.x > > Solr-Test-9.11 > > Solr-Test-9.x > > Solr-Test-main > > Solr-TestIntegration-9.11 > > Solr-TestIntegration-9.x > > > > On 2026/07/03 20:32:09 David Smiley wrote: > > > I know too little about Jenkins to know how to apply this solution, but I > > > can try to help. The Solr jobs are defined in Jenkins, not in our repo > > in > > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself? > > Does > > > this need to be done on all jobs or just some or will just one do? > > > > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <[email protected]> wrote: > > > > > > > This has come up as an issue again today, necessitating intervention to > > > > keep the service available. > > > > > > > > On 2026/06/29 07:04:23 Robert Thomson wrote: > > > > > Hi, > > > > > > > > > > We are regularly noticing a build-up of large ws-cleanup directories > > > > > on lucene3 and 4, necessitating deletion. Camel also experiences this > > > > > issue with builds and the root cause is that there are files created > > > > > as part of the docker build that are not owned by the user used for > > > > > the cleanup, so fail to be deleted by the workspace clean up plugin. > > > > > > > > > > Searching with GenAI came up with the following suggestion when > > > > > discussing with them that you may like to try out in your pipelines > > > > > that create docker images. > > > > > > > > > > Kind regards, > > > > > -Bob Thomson, > > > > > ASF Infrastructure > > > > > > > > > > post { > > > > > always { > > > > > // Fallback option: If standard cleanWs() stalls out on > > > > permissions > > > > > script { > > > > > try { > > > > > cleanWs deleteDirs: true, disableDeferredWipeout: > > true > > > > > } catch (Exception e) { > > > > > echo “Standard cleanup failed, launching Docker Root > > > > > Purge force-override...” > > > > > sh “”" > > > > > docker run --rm \ > > > > > -v ${WORKSPACE}/..:/zap \ > > > > > alpine:latest \ > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’ > > > > > “”" > > > > > } > > > > > } > > > > > } > > > > > } > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
