Yes, these are on the same volume in the /var/lib/docker directory. I'm unsure if they clean up leftover images.
On Fri, Jul 24, 2020 at 12:52 PM Udi Meiri <eh...@google.com> wrote: > I forgot Docker images: > > ehudm@apache-ci-beam-jenkins-3:~$ sudo docker system df > TYPE TOTAL ACTIVE SIZE > RECLAIMABLE > Images 88 9 125.4GB > 124.2GB (99%) > Containers 40 4 7.927GB > 7.871GB (99%) > Local Volumes 47 0 3.165GB > 3.165GB (100%) > Build Cache 0 0 0B > 0B > > There are about 90 images on that machine, with all but 1 less than 48 > hours old. > I think the docker test jobs need to try harder at cleaning up their > leftover images. (assuming they're already doing it?) > > On Fri, Jul 24, 2020 at 12:31 PM Udi Meiri <eh...@google.com> wrote: > >> The additional slots (@3 directories) take up even more space now than >> before. >> >> I'm testing out https://github.com/apache/beam/pull/12326 which could >> help by cleaning up workspaces after a run (just started a seed job). >> >> On Fri, Jul 24, 2020 at 12:13 PM Tyson Hamilton <tyso...@google.com> >> wrote: >> >>> 664M beam_PreCommit_JavaPortabilityApi_Commit >>> 656M beam_PreCommit_JavaPortabilityApi_Commit@2 >>> 611M beam_PreCommit_JavaPortabilityApi_Cron >>> 616M beam_PreCommit_JavaPortabilityApiJava11_Commit >>> 598M beam_PreCommit_JavaPortabilityApiJava11_Commit@2 >>> 662M beam_PreCommit_JavaPortabilityApiJava11_Cron >>> 2.9G beam_PreCommit_Portable_Python_Commit >>> 2.9G beam_PreCommit_Portable_Python_Commit@2 >>> 1.7G beam_PreCommit_Portable_Python_Commit@3 >>> 3.4G beam_PreCommit_Portable_Python_Cron >>> 1.9G beam_PreCommit_Python2_PVR_Flink_Commit >>> 1.4G beam_PreCommit_Python2_PVR_Flink_Cron >>> 1.3G beam_PreCommit_Python2_PVR_Flink_Phrase >>> 6.2G beam_PreCommit_Python_Commit >>> 7.5G beam_PreCommit_Python_Commit@2 >>> 7.5G beam_PreCommit_Python_Cron >>> 1012M beam_PreCommit_PythonDocker_Commit >>> 1011M beam_PreCommit_PythonDocker_Commit@2 >>> 1011M beam_PreCommit_PythonDocker_Commit@3 >>> 1002M beam_PreCommit_PythonDocker_Cron >>> 877M beam_PreCommit_PythonFormatter_Commit >>> 988M beam_PreCommit_PythonFormatter_Cron >>> 986M beam_PreCommit_PythonFormatter_Phrase >>> 1.7G beam_PreCommit_PythonLint_Commit >>> 2.1G beam_PreCommit_PythonLint_Cron >>> 7.5G beam_PreCommit_Python_Phrase >>> 346M beam_PreCommit_RAT_Commit >>> 341M beam_PreCommit_RAT_Cron >>> 338M beam_PreCommit_Spotless_Commit >>> 339M beam_PreCommit_Spotless_Cron >>> 5.5G beam_PreCommit_SQL_Commit >>> 5.5G beam_PreCommit_SQL_Cron >>> 5.5G beam_PreCommit_SQL_Java11_Commit >>> 750M beam_PreCommit_Website_Commit >>> 750M beam_PreCommit_Website_Commit@2 >>> 750M beam_PreCommit_Website_Cron >>> 764M beam_PreCommit_Website_Stage_GCS_Commit >>> 771M beam_PreCommit_Website_Stage_GCS_Cron >>> 336M beam_Prober_CommunityMetrics >>> 693M beam_python_mongoio_load_test >>> 339M beam_SeedJob >>> 333M beam_SeedJob_Standalone >>> 334M beam_sonarqube_report >>> 556M beam_SQLBigQueryIO_Batch_Performance_Test_Java >>> 175G total >>> >>> On Fri, Jul 24, 2020 at 12:04 PM Tyson Hamilton <tyso...@google.com> >>> wrote: >>> >>>> Ya looks like something in the workspaces is taking up room: >>>> >>>> @apache-ci-beam-jenkins-8:/home/jenkins$ sudo du -shc . >>>> 191G . >>>> 191G total >>>> >>>> >>>> On Fri, Jul 24, 2020 at 11:44 AM Tyson Hamilton <tyso...@google.com> >>>> wrote: >>>> >>>>> Node 8 is also full. The partition that /tmp is on is here: >>>>> >>>>> Filesystem Size Used Avail Use% Mounted on >>>>> /dev/sda1 485G 482G 2.9G 100% / >>>>> >>>>> however after cleaning up tmp with the crontab command, there is only >>>>> 8G usage yet it still remains 100% full: >>>>> >>>>> @apache-ci-beam-jenkins-8:/tmp$ sudo du -shc /tmp >>>>> 8.0G /tmp >>>>> 8.0G total >>>>> >>>>> The workspaces are in the /home/jenkins/jenkins-slave/workspace >>>>> directory. When I run a du on that, it takes really long. I'll let it keep >>>>> running for a while to see if it ever returns a result but so far this >>>>> seems suspect. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Jul 24, 2020 at 11:19 AM Tyson Hamilton <tyso...@google.com> >>>>> wrote: >>>>> >>>>>> Everything I've been looking at is in the /tmp dir. Where are the >>>>>> workspaces, or what are the named? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jul 24, 2020 at 11:03 AM Udi Meiri <eh...@google.com> wrote: >>>>>> >>>>>>> I'm curious to what you find. Was it /tmp or the workspaces using up >>>>>>> the space? >>>>>>> >>>>>>> On Fri, Jul 24, 2020 at 10:57 AM Tyson Hamilton <tyso...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Bleck. I just realized that it is 'offline' so that won't work. >>>>>>>> I'll clean up manually on the machine using the cron command. >>>>>>>> >>>>>>>> On Fri, Jul 24, 2020 at 10:56 AM Tyson Hamilton <tyso...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Something isn't working with the current set up because node 15 >>>>>>>>> appears to be out of space and is currently 'offline' according to >>>>>>>>> Jenkins. >>>>>>>>> Can someone run the cleanup job? The machine is full, >>>>>>>>> >>>>>>>>> @apache-ci-beam-jenkins-15:/tmp$ df -h >>>>>>>>> Filesystem Size Used Avail Use% Mounted on >>>>>>>>> udev 52G 0 52G 0% /dev >>>>>>>>> tmpfs 11G 265M 10G 3% /run >>>>>>>>> */dev/sda1 485G 484G 880M 100% /* >>>>>>>>> tmpfs 52G 0 52G 0% /dev/shm >>>>>>>>> tmpfs 5.0M 0 5.0M 0% /run/lock >>>>>>>>> tmpfs 52G 0 52G 0% /sys/fs/cgroup >>>>>>>>> tmpfs 11G 0 11G 0% /run/user/1017 >>>>>>>>> tmpfs 11G 0 11G 0% /run/user/1037 >>>>>>>>> >>>>>>>>> apache-ci-beam-jenkins-15:/tmp$ sudo du -ah --time . | sort -rhk >>>>>>>>> 1,1 | head -n 20 >>>>>>>>> 20G 2020-07-24 17:52 . >>>>>>>>> 580M 2020-07-22 17:31 ./junit1031982597110125586 >>>>>>>>> 517M 2020-07-22 17:31 >>>>>>>>> ./junit1031982597110125586/junit8739924829337821410/heap_dump.hprof >>>>>>>>> 517M 2020-07-22 17:31 >>>>>>>>> ./junit1031982597110125586/junit8739924829337821410 >>>>>>>>> 263M 2020-07-22 12:23 ./pip-install-2GUhO_ >>>>>>>>> 263M 2020-07-20 09:30 ./pip-install-sxgwqr >>>>>>>>> 263M 2020-07-17 13:56 ./pip-install-bWSKIV >>>>>>>>> 242M 2020-07-21 20:25 ./beam-pipeline-tempmByU6T >>>>>>>>> 242M 2020-07-21 20:21 ./beam-pipeline-tempV85xeK >>>>>>>>> 242M 2020-07-21 20:15 ./beam-pipeline-temp7dJROJ >>>>>>>>> 236M 2020-07-21 20:25 >>>>>>>>> ./beam-pipeline-tempmByU6T/tmpOWj3Yr >>>>>>>>> 236M 2020-07-21 20:21 >>>>>>>>> ./beam-pipeline-tempV85xeK/tmppbQHB3 >>>>>>>>> 236M 2020-07-21 20:15 >>>>>>>>> ./beam-pipeline-temp7dJROJ/tmpgOXPKW >>>>>>>>> 111M 2020-07-23 00:57 ./pip-install-1JnyNE >>>>>>>>> 105M 2020-07-23 00:17 ./beam-artifact1374651823280819755 >>>>>>>>> 105M 2020-07-23 00:16 ./beam-artifact5050755582921936972 >>>>>>>>> 105M 2020-07-23 00:16 ./beam-artifact1834064452502646289 >>>>>>>>> 105M 2020-07-23 00:15 ./beam-artifact682561790267074916 >>>>>>>>> 105M 2020-07-23 00:15 ./beam-artifact4691304965824489394 >>>>>>>>> 105M 2020-07-23 00:14 ./beam-artifact4050383819822604421 >>>>>>>>> >>>>>>>>> On Wed, Jul 22, 2020 at 12:03 PM Robert Bradshaw < >>>>>>>>> rober...@google.com> wrote: >>>>>>>>> >>>>>>>>>> On Wed, Jul 22, 2020 at 11:57 AM Tyson Hamilton < >>>>>>>>>> tyso...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> Ah I see, thanks Kenn. I found some advice from the Apache infra >>>>>>>>>>> wiki that also suggests using a tmpdir inside the workspace [1]: >>>>>>>>>>> >>>>>>>>>>> Procedures Projects can take to clean up disk space >>>>>>>>>>> >>>>>>>>>>> Projects can help themselves and Infra by taking some basic >>>>>>>>>>> steps to help clean up their jobs after themselves on the build >>>>>>>>>>> nodes. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 1. Use a ./tmp dir in your jobs workspace. That way it gets >>>>>>>>>>> cleaned up when job workspaces expire. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> Tests should be (able to be) written to use the standard >>>>>>>>>> temporary file mechanisms, and the environment set up on Jenkins >>>>>>>>>> such that >>>>>>>>>> that falls into the respective workspaces. Ideally this should be as >>>>>>>>>> simple >>>>>>>>>> as setting the TMPDIR (or similar) environment variable (and making >>>>>>>>>> sure it >>>>>>>>>> exists/is writable). >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 1. Configure your jobs to wipe workspaces on start or finish. >>>>>>>>>>> 2. Configure your jobs to only keep 5 or 10 previous builds. >>>>>>>>>>> 3. Configure your jobs to only keep 5 or 10 previous >>>>>>>>>>> artifacts. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [1]: >>>>>>>>>>> https://cwiki.apache.org/confluence/display/INFRA/Disk+Space+cleanup+of+Jenkins+nodes >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 22, 2020 at 8:06 AM Kenneth Knowles <k...@apache.org> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Those file listings look like the result of using standard temp >>>>>>>>>>>> file APIs but with TMPDIR set to /tmp. >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jul 20, 2020 at 7:55 PM Tyson Hamilton < >>>>>>>>>>>> tyso...@google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Jobs are hermetic as far as I can tell and use unique >>>>>>>>>>>>> subdirectories inside of /tmp. Here is a quick look into two >>>>>>>>>>>>> examples: >>>>>>>>>>>>> >>>>>>>>>>>>> @apache-ci-beam-jenkins-4:/tmp$ sudo du -ah --time . | sort >>>>>>>>>>>>> -rhk 1,1 | head -n 20 >>>>>>>>>>>>> 1.6G 2020-07-21 02:25 . >>>>>>>>>>>>> 242M 2020-07-17 18:48 ./beam-pipeline-temp3ybuY4 >>>>>>>>>>>>> 242M 2020-07-17 18:46 ./beam-pipeline-tempuxjiPT >>>>>>>>>>>>> 242M 2020-07-17 18:44 ./beam-pipeline-tempVpg1ME >>>>>>>>>>>>> 242M 2020-07-17 18:42 ./beam-pipeline-tempJ4EpyB >>>>>>>>>>>>> 242M 2020-07-17 18:39 ./beam-pipeline-tempepea7Q >>>>>>>>>>>>> 242M 2020-07-17 18:35 ./beam-pipeline-temp79qot2 >>>>>>>>>>>>> 236M 2020-07-17 18:48 >>>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmpy_Ytzz >>>>>>>>>>>>> 236M 2020-07-17 18:46 >>>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpN5_UfJ >>>>>>>>>>>>> 236M 2020-07-17 18:44 >>>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpxSm8pX >>>>>>>>>>>>> 236M 2020-07-17 18:42 >>>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpMZJU76 >>>>>>>>>>>>> 236M 2020-07-17 18:39 >>>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmpWy1vWX >>>>>>>>>>>>> 236M 2020-07-17 18:35 >>>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmpvN7vWA >>>>>>>>>>>>> 3.7M 2020-07-17 18:48 >>>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmprlh_di >>>>>>>>>>>>> 3.7M 2020-07-17 18:46 >>>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpLmVWfe >>>>>>>>>>>>> 3.7M 2020-07-17 18:44 >>>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpvrxbY7 >>>>>>>>>>>>> 3.7M 2020-07-17 18:42 >>>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpLTb6Mj >>>>>>>>>>>>> 3.7M 2020-07-17 18:39 >>>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmptYF1v1 >>>>>>>>>>>>> 3.7M 2020-07-17 18:35 >>>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmplfV0Rg >>>>>>>>>>>>> 2.7M 2020-07-17 20:10 ./pip-install-q9l227ef >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> @apache-ci-beam-jenkins-11:/tmp$ sudo du -ah --time . | sort >>>>>>>>>>>>> -rhk 1,1 | head -n 20 >>>>>>>>>>>>> 817M 2020-07-21 02:26 . >>>>>>>>>>>>> 242M 2020-07-19 12:14 ./beam-pipeline-tempUTXqlM >>>>>>>>>>>>> 242M 2020-07-19 12:11 ./beam-pipeline-tempx3Yno3 >>>>>>>>>>>>> 242M 2020-07-19 12:05 ./beam-pipeline-tempyCrMYq >>>>>>>>>>>>> 236M 2020-07-19 12:14 >>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpstXoL0 >>>>>>>>>>>>> 236M 2020-07-19 12:11 >>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpnnVn65 >>>>>>>>>>>>> 236M 2020-07-19 12:05 >>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpRF0iNs >>>>>>>>>>>>> 3.7M 2020-07-19 12:14 >>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpbJjUAQ >>>>>>>>>>>>> 3.7M 2020-07-19 12:11 >>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpsmmzqe >>>>>>>>>>>>> 3.7M 2020-07-19 12:05 >>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmp5b3ZvY >>>>>>>>>>>>> 2.0M 2020-07-19 12:14 >>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpoj3orz >>>>>>>>>>>>> 2.0M 2020-07-19 12:11 >>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmptng9sZ >>>>>>>>>>>>> 2.0M 2020-07-19 12:05 >>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpWp6njc >>>>>>>>>>>>> 1.2M 2020-07-19 12:14 >>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmphgdj35 >>>>>>>>>>>>> 1.2M 2020-07-19 12:11 >>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmp8ySXpm >>>>>>>>>>>>> 1.2M 2020-07-19 12:05 >>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpNVEJ4e >>>>>>>>>>>>> 992K 2020-07-12 12:00 ./junit642086915811430564 >>>>>>>>>>>>> 988K 2020-07-12 12:00 ./junit642086915811430564/beam >>>>>>>>>>>>> 984K 2020-07-12 12:00 >>>>>>>>>>>>> ./junit642086915811430564/beam/nodes >>>>>>>>>>>>> 980K 2020-07-12 12:00 >>>>>>>>>>>>> ./junit642086915811430564/beam/nodes/0 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jul 20, 2020 at 6:46 PM Udi Meiri <eh...@google.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> You're right, job workspaces should be hermetic. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 1:24 PM Kenneth Knowles < >>>>>>>>>>>>>> k...@apache.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm probably late to this discussion and missing something, >>>>>>>>>>>>>>> but why are we writing to /tmp at all? I would expect TMPDIR to >>>>>>>>>>>>>>> point >>>>>>>>>>>>>>> somewhere inside the job directory that will be wiped by >>>>>>>>>>>>>>> Jenkins, and I >>>>>>>>>>>>>>> would expect code to always create temp files via APIs that >>>>>>>>>>>>>>> respect this. >>>>>>>>>>>>>>> Is Jenkins not cleaning up? Do we not have the ability to set >>>>>>>>>>>>>>> this up? Do >>>>>>>>>>>>>>> we have bugs in our code (that we could probably find by >>>>>>>>>>>>>>> setting TMPDIR to >>>>>>>>>>>>>>> somewhere not-/tmp and running the tests without write >>>>>>>>>>>>>>> permission to /tmp, >>>>>>>>>>>>>>> etc) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kenn >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 11:39 AM Ahmet Altay < >>>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Related to workspace directory growth, +Udi Meiri >>>>>>>>>>>>>>>> <eh...@google.com> filed a relevant issue previously ( >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-9865) for >>>>>>>>>>>>>>>> cleaning up workspace directory after successful jobs. >>>>>>>>>>>>>>>> Alternatively, we >>>>>>>>>>>>>>>> can consider periodically cleaning up the /src directories. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would suggest moving the cron task from internal cron >>>>>>>>>>>>>>>> scripts to the inventory job ( >>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L51). >>>>>>>>>>>>>>>> That way, we can see all the cron jobs as part of the source >>>>>>>>>>>>>>>> tree, adjust >>>>>>>>>>>>>>>> frequencies and clean up codes with PRs. I do not know how >>>>>>>>>>>>>>>> internal cron >>>>>>>>>>>>>>>> scripts are created, maintained, and how would they be >>>>>>>>>>>>>>>> recreated for new >>>>>>>>>>>>>>>> worker instances. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> /cc +Tyson Hamilton <tyso...@google.com> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 4:50 AM Damian Gadomski < >>>>>>>>>>>>>>>> damian.gadom...@polidea.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hey, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've recently created a solution for the growing /tmp >>>>>>>>>>>>>>>>> directory. Part of it is the job mentioned by Tyson: >>>>>>>>>>>>>>>>> *beam_Clean_tmp_directory*. It's intentionally not >>>>>>>>>>>>>>>>> triggered by cron and should be a last resort solution for >>>>>>>>>>>>>>>>> some strange >>>>>>>>>>>>>>>>> cases. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Along with that job, I've also updated every worker with >>>>>>>>>>>>>>>>> an internal cron script. It's being executed once a week and >>>>>>>>>>>>>>>>> deletes all >>>>>>>>>>>>>>>>> the files (and only files) that were not accessed for at >>>>>>>>>>>>>>>>> least three days. >>>>>>>>>>>>>>>>> That's designed to be as safe as possible for the running >>>>>>>>>>>>>>>>> jobs on the >>>>>>>>>>>>>>>>> worker (not to delete the files that are still in use), and >>>>>>>>>>>>>>>>> also to be >>>>>>>>>>>>>>>>> insensitive to the current workload on the machine. The >>>>>>>>>>>>>>>>> cleanup will always >>>>>>>>>>>>>>>>> happen, even if some long-running/stuck jobs are blocking the >>>>>>>>>>>>>>>>> machine. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I also think that currently the "No space left" errors >>>>>>>>>>>>>>>>> may be a consequence of growing workspace directory rather >>>>>>>>>>>>>>>>> than /tmp. I >>>>>>>>>>>>>>>>> didn't do any detailed analysis but e.g. currently, on >>>>>>>>>>>>>>>>> apache-beam-jenkins-7 the workspace directory size is 158 GB >>>>>>>>>>>>>>>>> while /tmp is >>>>>>>>>>>>>>>>> only 16 GB. We should either guarantee the disk size to hold >>>>>>>>>>>>>>>>> workspaces for >>>>>>>>>>>>>>>>> all jobs (because eventually, every worker will execute each >>>>>>>>>>>>>>>>> job) or clear >>>>>>>>>>>>>>>>> also the workspaces in some way. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Damian >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 10:43 AM Maximilian Michels < >>>>>>>>>>>>>>>>> m...@apache.org> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +1 for scheduling it via a cron job if it won't lead to >>>>>>>>>>>>>>>>>> test failures >>>>>>>>>>>>>>>>>> while running. Not a Jenkins expert but maybe there is >>>>>>>>>>>>>>>>>> the notion of >>>>>>>>>>>>>>>>>> running exclusively while no other tasks are running? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -Max >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 17.07.20 21:49, Tyson Hamilton wrote: >>>>>>>>>>>>>>>>>> > FYI there was a job introduced to do this in Jenkins: >>>>>>>>>>>>>>>>>> beam_Clean_tmp_directory >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > Currently it needs to be run manually. I'm seeing some >>>>>>>>>>>>>>>>>> out of disk related errors in precommit tests currently, >>>>>>>>>>>>>>>>>> perhaps we should >>>>>>>>>>>>>>>>>> schedule this job with cron? >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > On 2020/03/11 19:31:13, Heejong Lee <heej...@google.com> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >> Still seeing no space left on device errors on >>>>>>>>>>>>>>>>>> jenkins-7 (for example: >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2754/ >>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> On Fri, Mar 6, 2020 at 7:11 PM Alan Myrvold < >>>>>>>>>>>>>>>>>> amyrv...@google.com> wrote: >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >>> Did a one time cleanup of tmp files owned by jenkins >>>>>>>>>>>>>>>>>> older than 3 days. >>>>>>>>>>>>>>>>>> >>> Agree that we need a longer term solution. >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> Passing recent tests on all executors except >>>>>>>>>>>>>>>>>> jenkins-12, which has not >>>>>>>>>>>>>>>>>> >>> scheduled recent builds for the past 13 days. Not >>>>>>>>>>>>>>>>>> scheduling: >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-12/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-12/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> Recent passing builds: >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-1/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-1/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-2/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-2/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-3/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-3/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-4/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-4/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-5/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-5/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-6/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-6/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-7/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-7/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-8/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-8/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-9/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-9/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-10/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-10/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-11/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-11/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-13/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-13/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-14/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-14/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-15/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-15/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-16/builds >>>>>>>>>>>>>>>>>> >>> < >>>>>>>>>>>>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-16/builds&sa=D >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>> On Fri, Mar 6, 2020 at 11:54 AM Ahmet Altay < >>>>>>>>>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>> >>>> +Alan Myrvold <amyrv...@google.com> is doing a one >>>>>>>>>>>>>>>>>> time cleanup. I agree >>>>>>>>>>>>>>>>>> >>>> that we need to have a solution to automate this >>>>>>>>>>>>>>>>>> task or address the root >>>>>>>>>>>>>>>>>> >>>> cause of the buildup. >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>> On Thu, Mar 5, 2020 at 2:47 AM Michał Walenia < >>>>>>>>>>>>>>>>>> michal.wale...@polidea.com> >>>>>>>>>>>>>>>>>> >>>> wrote: >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >>>>> Hi there, >>>>>>>>>>>>>>>>>> >>>>> it seems we have a problem with Jenkins workers >>>>>>>>>>>>>>>>>> again. Nodes 1 and 7 >>>>>>>>>>>>>>>>>> >>>>> both fail jobs with "No space left on device". >>>>>>>>>>>>>>>>>> >>>>> Who is the best person to contact in these cases >>>>>>>>>>>>>>>>>> (someone with access >>>>>>>>>>>>>>>>>> >>>>> permissions to the workers). >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> I also noticed that such errors are becoming more >>>>>>>>>>>>>>>>>> and more frequent >>>>>>>>>>>>>>>>>> >>>>> recently and I'd like to discuss how can this be >>>>>>>>>>>>>>>>>> remedied. Can a cleanup >>>>>>>>>>>>>>>>>> >>>>> task be automated on Jenkins somehow? >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> Regards >>>>>>>>>>>>>>>>>> >>>>> Michal >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> -- >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> Michał Walenia >>>>>>>>>>>>>>>>>> >>>>> Polidea <https://www.polidea.com/> | Software >>>>>>>>>>>>>>>>>> Engineer >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> M: +48 791 432 002 <+48%20791%20432%20002> < >>>>>>>>>>>>>>>>>> +48791432002 <+48%20791%20432%20002>> >>>>>>>>>>>>>>>>>> >>>>> E: michal.wale...@polidea.com >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>>> Unique Tech >>>>>>>>>>>>>>>>>> >>>>> Check out our projects! < >>>>>>>>>>>>>>>>>> https://www.polidea.com/our-work> >>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>
