This can also happen with a misconfigured logrotate config. If a process is writing to a large log file, and logrotate comes along and removes it, then the process still has an open filehandle to the large file even though you can't see it. The space won't get removed until the process closes the filehandle (eg rebooting).

The following command should give you a list of these ghost files that are still open but have been removed from the directory tree:

lsof | grep '(deleted)'

Stackexchange has some additional useful tips:

https://unix.stackexchange.com/questions/68523/find-and-remove-large-files-that-are-open-but-have-been-deleted

--Mike

On 6/22/22 15:49, matthew.st...@fujitsu.com wrote:
Deleted some files to "clean up" /var/log, but the space was not recovered?

Space for deleted files are only recovered when all references to that space 
are removed.   This includes the directory reference, and any open file-handles 
to the file.

The is a popular trick for a self-deleting scratch file.  Create a file, open 
file-handle to it, and then remove the file.   The open file-handle can then be 
written to, and read from, as long as you like, but when the file-handle is 
closed, explicitly or implicitly, the space is automatically recovered.


-----Original Message-----
From: Andrei Verovski <andre...@starlett.lv>
Sent: Wednesday, June 22, 2022 3:27 PM
To: users@ovirt.org
Subject: [ovirt-users] Lost space in /var/log

Hi !

I have strange situation with low disk space on /var/log/, yet I can't figure 
out what have consumed so much space.
Look at the du output.

Thanks in advance for any suggestion(s).


# df -h

Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos-var           15G  1.3G   13G   9% /var
/dev/mapper/centos-var_log       7.8G  7.1G  293M  97% /var/log
/dev/mapper/centos-var_log_audit 2.0G   41M  1.8G   3% /var/log/audit

#
# du -ah /var/log | sort -n -r | head -n 20

664K    /var/log/vdsm
660K    /var/log/vdsm/vdsm.log
624K    /var/log/gdm
580K    /var/log/anaconda/storage.log
480K    /var/log/anaconda/packaging.log
380K    /var/log/gdm/:0.log.4
316K    /var/log/anaconda/syslog
276K    /var/log/tuned
252K    /var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806
220K    /var/log/Xorg.0.log
168K    /var/log/gdm/:0.log
156K    /var/log/yum.log-20191117
156K    /var/log/secure-20180726
132K    /var/log/libvirt/qemu/NextCloud-Collabora-Active.log
128K    /var/log/anaconda/anaconda.log
120K    /var/log/hp-snmp-agents
116K    /var/log/hp-snmp-agents/cma.log
112K    /var/log/vinchin
104K    /var/log/vinchin/kvm_backup_service
104K    /var/log/tuned/tuned.log.2

#
# find . -printf '%s %p\n'| sort -nr | head -10

16311686 ./rhsm/rhsm.log
4070423 ./cron-20180726
3667146 ./anaconda/journal.log
3409071 ./secure
3066660 ./rhsm/rhsm.log-20180726
2912670 ./audit/audit.log
1418007 ./sanlock.log
1189580 ./vdsm/vdsm.log
592718 ./anaconda/storage.log
487567 ./anaconda/packaging.log
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXILL77BBB2EYGTFX7ZQ2WY/
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JCAQU32UZSEWKYRICOTC6QJMSC5BUZIW/
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HKTYSWZ4D3ZLBZ4LYIC4KTUC3NHD6YBS/

Reply via email to