I have a job that needs at least 800GiB of free disk space (per matrix sub-configuration) to build on a node. The majority of the space is needed to checkout git-lfs repos that contain test data. As cloning the lfs repos takes hours, the workspace is intentionally not cleaned up between builds. This job is fairly CPU intensive and may take over 10 hours, depending on the configuration. It is also only run once a day, so other jobs are allowed to build on the same nodes instead of having multiple build agents with a < 50% duty cycle. Nodes are taken offline when they are down to 50GiB of free disk and are returned to service by a cleanup script that has a "cron" trigger. The problem I occasionally run into is that a node has considerably more than the 50GiB offline/cleanup threshold, it does not have a pre-existing workspace for the heavy disk job, and has less disk available than the ~800GiB needed to build from scratch.
Ideally, from a job/build scheduling point of view, this job would have metadata declaring that it needs ~800GiB of diskspace space -- inclusive of the footprint of a pre-existing workspace. I have been unable to identify a method to achieve this via plugins (DiskCheck/DiskUsage, Job Restrictions, etc.). I understand that tracking per build/workspace disk usage is expensive, so that may simply not be a viable scheme. Completely dedicating multiple nodes to this job isn't feasible for me in terms of resource usage. Booting/tearing down VMs specifically for this job may be an option. Another possibility would be to mount additional block storage and running the job in a customworkspace (but I'm concerned would break knowing that this job was out of diskspace). Has anyone tackled this problem? Cheers, -Josh -- -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/997488fa-0768-5eeb-2801-d5f7e11a3989%40hoblitt.com. For more options, visit https://groups.google.com/d/optout.
