[ovirt-users] Re: Lost space in /var/log

2022-06-22 Thread wodel youchi
Have tried to reboot the node?

On Wed, Jun 22, 2022, 23:30 Andrei Verovski  wrote:

> Hi,
>
> Version 4.4
> Its simple 3-node system with no hosted engine (engine on separate server).
> Node installed manually, not via factory image.
> I just can't any file/dir which eats almost 8 GB on /var/log.
>
>
> On 6/22/22 23:40, wodel youchi wrote:
> > Is it an HCI deployment? I have had a similar problem and it was on an
> > HCI deployment. The healing process get stuck for some reason that I
> > don't know and it fills the space. What I did is reboot the node and
> > the free space was there.
> >
> > What version of ovirt are you using?
> >
> > Regards
> >
> > On Wed, Jun 22, 2022, 21:27 Andrei Verovski 
> wrote:
> >
> > Hi !
> >
> > I have strange situation with low disk space on /var/log/, yet I
> > can't
> > figure out what have consumed so much space.
> > Look at the du output.
> >
> > Thanks in advance for any suggestion(s).
> >
> >
> > # df -h
> >
> > Filesystem   Size  Used Avail Use% Mounted on
> > /dev/mapper/centos-var   15G  1.3G   13G   9% /var
> > /dev/mapper/centos-var_log   7.8G  7.1G  293M  97% /var/log
> > /dev/mapper/centos-var_log_audit 2.0G   41M  1.8G   3% /var/log/audit
> >
> > #
> > # du -ah /var/log | sort -n -r | head -n 20
> >
> > 664K/var/log/vdsm
> > 660K/var/log/vdsm/vdsm.log
> > 624K/var/log/gdm
> > 580K/var/log/anaconda/storage.log
> > 480K/var/log/anaconda/packaging.log
> > 380K/var/log/gdm/:0.log.4
> > 316K/var/log/anaconda/syslog
> > 276K/var/log/tuned
> > 252K /var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806
> > 220K/var/log/Xorg.0.log
> > 168K/var/log/gdm/:0.log
> > 156K/var/log/yum.log-20191117
> > 156K/var/log/secure-20180726
> > 132K/var/log/libvirt/qemu/NextCloud-Collabora-Active.log
> > 128K/var/log/anaconda/anaconda.log
> > 120K/var/log/hp-snmp-agents
> > 116K/var/log/hp-snmp-agents/cma.log
> > 112K/var/log/vinchin
> > 104K/var/log/vinchin/kvm_backup_service
> > 104K/var/log/tuned/tuned.log.2
> >
> > #
> > # find . -printf '%s %p\n'| sort -nr | head -10
> >
> > 16311686 ./rhsm/rhsm.log
> > 4070423 ./cron-20180726
> > 3667146 ./anaconda/journal.log
> > 3409071 ./secure
> > 300 ./rhsm/rhsm.log-20180726
> > 2912670 ./audit/audit.log
> > 1418007 ./sanlock.log
> > 1189580 ./vdsm/vdsm.log
> > 592718 ./anaconda/storage.log
> > 487567 ./anaconda/packaging.log
> > ___
> > Users mailing list -- users@ovirt.org
> > To unsubscribe send an email to users-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> >
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXILL77BBB2EYGTFX7ZQ2WY/
> >
>
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2LT3FDUVZMWZ7GGTJA4M3AFD4HTNOJSW/


[ovirt-users] Re: Lost space in /var/log

2022-06-22 Thread Andrei Verovski

Hi,

Version 4.4
Its simple 3-node system with no hosted engine (engine on separate server).
Node installed manually, not via factory image.
I just can't any file/dir which eats almost 8 GB on /var/log.


On 6/22/22 23:40, wodel youchi wrote:
Is it an HCI deployment? I have had a similar problem and it was on an 
HCI deployment. The healing process get stuck for some reason that I 
don't know and it fills the space. What I did is reboot the node and 
the free space was there.


What version of ovirt are you using?

Regards

On Wed, Jun 22, 2022, 21:27 Andrei Verovski  wrote:

Hi !

I have strange situation with low disk space on /var/log/, yet I
can't
figure out what have consumed so much space.
Look at the du output.

Thanks in advance for any suggestion(s).


# df -h

Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos-var           15G  1.3G   13G   9% /var
/dev/mapper/centos-var_log       7.8G  7.1G  293M  97% /var/log
/dev/mapper/centos-var_log_audit 2.0G   41M  1.8G   3% /var/log/audit

#
# du -ah /var/log | sort -n -r | head -n 20

664K    /var/log/vdsm
660K    /var/log/vdsm/vdsm.log
624K    /var/log/gdm
580K    /var/log/anaconda/storage.log
480K    /var/log/anaconda/packaging.log
380K    /var/log/gdm/:0.log.4
316K    /var/log/anaconda/syslog
276K    /var/log/tuned
252K /var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806
220K    /var/log/Xorg.0.log
168K    /var/log/gdm/:0.log
156K    /var/log/yum.log-20191117
156K    /var/log/secure-20180726
132K    /var/log/libvirt/qemu/NextCloud-Collabora-Active.log
128K    /var/log/anaconda/anaconda.log
120K    /var/log/hp-snmp-agents
116K    /var/log/hp-snmp-agents/cma.log
112K    /var/log/vinchin
104K    /var/log/vinchin/kvm_backup_service
104K    /var/log/tuned/tuned.log.2

#
# find . -printf '%s %p\n'| sort -nr | head -10

16311686 ./rhsm/rhsm.log
4070423 ./cron-20180726
3667146 ./anaconda/journal.log
3409071 ./secure
300 ./rhsm/rhsm.log-20180726
2912670 ./audit/audit.log
1418007 ./sanlock.log
1189580 ./vdsm/vdsm.log
592718 ./anaconda/storage.log
487567 ./anaconda/packaging.log
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:

https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXILL77BBB2EYGTFX7ZQ2WY/


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/IUQK5ZCEDH7IIKUSFL2BI2V4KIWREBBS/


[ovirt-users] Re: Lost space in /var/log

2022-06-22 Thread Michael Thomas
This can also happen with a misconfigured logrotate config.  If a 
process is writing to a large log file, and logrotate comes along and 
removes it, then the process still has an open filehandle to the large 
file even though you can't see it.  The space won't get removed until 
the process closes the filehandle (eg rebooting).


The following command should give you a list of these ghost files that 
are still open but have been removed from the directory tree:


lsof | grep '(deleted)'

Stackexchange has some additional useful tips:

https://unix.stackexchange.com/questions/68523/find-and-remove-large-files-that-are-open-but-have-been-deleted

--Mike

On 6/22/22 15:49, matthew.st...@fujitsu.com wrote:

Deleted some files to "clean up" /var/log, but the space was not recovered?

Space for deleted files are only recovered when all references to that space 
are removed.   This includes the directory reference, and any open file-handles 
to the file.

The is a popular trick for a self-deleting scratch file.  Create a file, open 
file-handle to it, and then remove the file.   The open file-handle can then be 
written to, and read from, as long as you like, but when the file-handle is 
closed, explicitly or implicitly, the space is automatically recovered.


-Original Message-
From: Andrei Verovski 
Sent: Wednesday, June 22, 2022 3:27 PM
To: users@ovirt.org
Subject: [ovirt-users] Lost space in /var/log

Hi !

I have strange situation with low disk space on /var/log/, yet I can't figure 
out what have consumed so much space.
Look at the du output.

Thanks in advance for any suggestion(s).


# df -h

Filesystem   Size  Used Avail Use% Mounted on
/dev/mapper/centos-var   15G  1.3G   13G   9% /var
/dev/mapper/centos-var_log   7.8G  7.1G  293M  97% /var/log
/dev/mapper/centos-var_log_audit 2.0G   41M  1.8G   3% /var/log/audit

#
# du -ah /var/log | sort -n -r | head -n 20

664K/var/log/vdsm
660K/var/log/vdsm/vdsm.log
624K/var/log/gdm
580K/var/log/anaconda/storage.log
480K/var/log/anaconda/packaging.log
380K/var/log/gdm/:0.log.4
316K/var/log/anaconda/syslog
276K/var/log/tuned
252K/var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806
220K/var/log/Xorg.0.log
168K/var/log/gdm/:0.log
156K/var/log/yum.log-20191117
156K/var/log/secure-20180726
132K/var/log/libvirt/qemu/NextCloud-Collabora-Active.log
128K/var/log/anaconda/anaconda.log
120K/var/log/hp-snmp-agents
116K/var/log/hp-snmp-agents/cma.log
112K/var/log/vinchin
104K/var/log/vinchin/kvm_backup_service
104K/var/log/tuned/tuned.log.2

#
# find . -printf '%s %p\n'| sort -nr | head -10

16311686 ./rhsm/rhsm.log
4070423 ./cron-20180726
3667146 ./anaconda/journal.log
3409071 ./secure
300 ./rhsm/rhsm.log-20180726
2912670 ./audit/audit.log
1418007 ./sanlock.log
1189580 ./vdsm/vdsm.log
592718 ./anaconda/storage.log
487567 ./anaconda/packaging.log
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXILL77BBB2EYGTFX7ZQ2WY/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JCAQU32UZSEWKYRICOTC6QJMSC5BUZIW/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HKTYSWZ4D3ZLBZ4LYIC4KTUC3NHD6YBS/


[ovirt-users] Re: Lost space in /var/log

2022-06-22 Thread matthew.st...@fujitsu.com
Deleted some files to "clean up" /var/log, but the space was not recovered?

Space for deleted files are only recovered when all references to that space 
are removed.   This includes the directory reference, and any open file-handles 
to the file.

The is a popular trick for a self-deleting scratch file.  Create a file, open 
file-handle to it, and then remove the file.   The open file-handle can then be 
written to, and read from, as long as you like, but when the file-handle is 
closed, explicitly or implicitly, the space is automatically recovered.


-Original Message-
From: Andrei Verovski  
Sent: Wednesday, June 22, 2022 3:27 PM
To: users@ovirt.org
Subject: [ovirt-users] Lost space in /var/log

Hi !

I have strange situation with low disk space on /var/log/, yet I can't figure 
out what have consumed so much space.
Look at the du output.

Thanks in advance for any suggestion(s).


# df -h

Filesystem   Size  Used Avail Use% Mounted on
/dev/mapper/centos-var   15G  1.3G   13G   9% /var
/dev/mapper/centos-var_log   7.8G  7.1G  293M  97% /var/log
/dev/mapper/centos-var_log_audit 2.0G   41M  1.8G   3% /var/log/audit

#
# du -ah /var/log | sort -n -r | head -n 20

664K/var/log/vdsm
660K/var/log/vdsm/vdsm.log
624K/var/log/gdm
580K/var/log/anaconda/storage.log
480K/var/log/anaconda/packaging.log
380K/var/log/gdm/:0.log.4
316K/var/log/anaconda/syslog
276K/var/log/tuned
252K/var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806
220K/var/log/Xorg.0.log
168K/var/log/gdm/:0.log
156K/var/log/yum.log-20191117
156K/var/log/secure-20180726
132K/var/log/libvirt/qemu/NextCloud-Collabora-Active.log
128K/var/log/anaconda/anaconda.log
120K/var/log/hp-snmp-agents
116K/var/log/hp-snmp-agents/cma.log
112K/var/log/vinchin
104K/var/log/vinchin/kvm_backup_service
104K/var/log/tuned/tuned.log.2

#
# find . -printf '%s %p\n'| sort -nr | head -10

16311686 ./rhsm/rhsm.log
4070423 ./cron-20180726
3667146 ./anaconda/journal.log
3409071 ./secure
300 ./rhsm/rhsm.log-20180726
2912670 ./audit/audit.log
1418007 ./sanlock.log
1189580 ./vdsm/vdsm.log
592718 ./anaconda/storage.log
487567 ./anaconda/packaging.log
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: 
https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXILL77BBB2EYGTFX7ZQ2WY/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/JCAQU32UZSEWKYRICOTC6QJMSC5BUZIW/


[ovirt-users] Re: Lost space in /var/log

2022-06-22 Thread wodel youchi
Is it an HCI deployment? I have had a similar problem and it was on an HCI
deployment. The healing process get stuck for some reason that I don't know
and it fills the space. What I did is reboot the node and the free space
was there.

What version of ovirt are you using?

Regards

On Wed, Jun 22, 2022, 21:27 Andrei Verovski  wrote:

> Hi !
>
> I have strange situation with low disk space on /var/log/, yet I can't
> figure out what have consumed so much space.
> Look at the du output.
>
> Thanks in advance for any suggestion(s).
>
>
> # df -h
>
> Filesystem   Size  Used Avail Use% Mounted on
> /dev/mapper/centos-var   15G  1.3G   13G   9% /var
> /dev/mapper/centos-var_log   7.8G  7.1G  293M  97% /var/log
> /dev/mapper/centos-var_log_audit 2.0G   41M  1.8G   3% /var/log/audit
>
> #
> # du -ah /var/log | sort -n -r | head -n 20
>
> 664K/var/log/vdsm
> 660K/var/log/vdsm/vdsm.log
> 624K/var/log/gdm
> 580K/var/log/anaconda/storage.log
> 480K/var/log/anaconda/packaging.log
> 380K/var/log/gdm/:0.log.4
> 316K/var/log/anaconda/syslog
> 276K/var/log/tuned
> 252K/var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806
> 220K/var/log/Xorg.0.log
> 168K/var/log/gdm/:0.log
> 156K/var/log/yum.log-20191117
> 156K/var/log/secure-20180726
> 132K/var/log/libvirt/qemu/NextCloud-Collabora-Active.log
> 128K/var/log/anaconda/anaconda.log
> 120K/var/log/hp-snmp-agents
> 116K/var/log/hp-snmp-agents/cma.log
> 112K/var/log/vinchin
> 104K/var/log/vinchin/kvm_backup_service
> 104K/var/log/tuned/tuned.log.2
>
> #
> # find . -printf '%s %p\n'| sort -nr | head -10
>
> 16311686 ./rhsm/rhsm.log
> 4070423 ./cron-20180726
> 3667146 ./anaconda/journal.log
> 3409071 ./secure
> 300 ./rhsm/rhsm.log-20180726
> 2912670 ./audit/audit.log
> 1418007 ./sanlock.log
> 1189580 ./vdsm/vdsm.log
> 592718 ./anaconda/storage.log
> 487567 ./anaconda/packaging.log
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXILL77BBB2EYGTFX7ZQ2WY/
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KANNWUKYAC5IR6ZQ56XRSAV6XU4IE6G7/