Stamatis Zampetakis created HIVE-28359:
------------------------------------------
Summary: Discard old builds in Jenkins to avoid disk space
exhaustion
Key: HIVE-28359
URL: https://issues.apache.org/jira/browse/HIVE-28359
Project: Hive
Issue Type: Task
Components: Testing Infrastructure
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
Attachments: builds.txt
Currently Jenkins retains the builds from all active branches/PRs.
{code:bash}
for b in `find var/jenkins_home/jobs -name "builds"`; do echo -n $b" " ; ls -l
$b | wc -l; done | sort -k2 -rn > builds.txt
{code}
Some PRs (e.g.,
[PR-5216|https://ci.hive.apache.org/job/hive-precommit/view/change-requests/job/PR-5216/])
with an excessive number of builds (i.e., 66) can easily consume many GBs of
data (PR-5216 uses 13GB for the builds). The first build for PR-5216 was saved
on April 26, 2024 and it is now more than 2 months old.
For master, we currently have all builds since January 2023 (previous builds
where manually removed as part of HIVE-28013). The builds for master occupy
currently 50GB of space.
Due to the above the disk space (persistent volume) cannot be reclaimed and
currently it is almost full (91% /var/jenkins_home).
{noformat}
kubectl exec jenkins-6858ddb664-l4xfg -- bash -c "df"
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 98831908 4675004 94140520 5% /
tmpfs 65536 0 65536 0% /dev
tmpfs 6645236 0 6645236 0% /sys/fs/cgroup
/dev/sdb 308521792 278996208 29509200 91% /var/jenkins_home
/dev/sda1 98831908 4675004 94140520 5% /etc/hosts
shm 65536 0 65536 0% /dev/shm
tmpfs 10801128 12 10801116 1%
/run/secrets/kubernetes.io/serviceaccount
tmpfs 6645236 0 6645236 0% /proc/acpi
tmpfs 6645236 0 6645236 0% /proc/scsi
tmpfs 6645236 0 6645236 0% /sys/firmware
{noformat}
Without a discard policy in place we are going to hit again HIVE-28013 or other
disk related issues pretty soon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
