Hi,
I found a couple of ways to empty out an existing large file, using Output
redirection " > filename " or "truncate -s 0 filename ".
I checked after emptying the file and I see the logs continue normally.
This way, I don't need to delete the pod or kill a process.
root@olc-bjng-3:/var/lib/docker/containers# du -h -d 1 | sort -r -h | head
49G .
35G ./b9df9b8587860364bc4267889f81712544e5aa6ce3e5ab33c8c3842fd014c410
14G ./ad21fbc6614f291c50838e860587f568f45eea1956408f1ec078bb08fcfd5879
root@olc-bjng-3:/var/lib/docker/containers# >
./b9df9b8587860364bc4267889f81712544e5aa6ce3e5ab33c8c3842fd014c410/b9df9b8587860364bc4267889f81712544e5aa6ce3e5ab33c8c3842fd014c410-json.log
root@olc-bjng-3:/var/lib/docker/containers/b9df9b8587860364bc4267889f81712544e5aa6ce3e5ab33c8c3842fd014c410#
ls -l
total 60
-rw-r----- 1 root root 193 Jun 13
20:52b9df9b8587860364bc4267889f81712544e5aa6ce3e5ab33c8c3842fd014c410-json.log
drwx------ 2 root root 4096 Jun 11 15:41checkpoints
-rw-r--r-- 1 root root 48062 Jun 11 15:41config.v2.json
-rw-r--r-- 1 root root 1696 Jun 11 15:41hostconfig.json
root@olc-bjng-3:/var/lib/docker/containers/b9df9b8587860364bc4267889f81712544e5aa6ce3e5ab33c8c3842fd014c410#
cat b9df9b8587860364bc4267889f81712544e5aa6ce3e5ab33c8c3842fd014c410-json.log
{"log":"2018-06-13 20:52:12.925 INFO 451 --- [ main]
o.o.u.server.util.DmaapSubscriber : response content is
:[]\n","stream":"stdout","time":"2018-06-13T20:52:12.932234368Z"}
{"log":"2018-06-13 20:52:23.078 INFO 451 --- [ main]
o.o.u.server.util.DmaapSubscriber : response content is
:[]\n","stream":"stdout","time":"2018-06-13T20:52:23.084544646Z"}
{"log":"2018-06-13 20:52:33.217 INFO 451 --- [ main]
o.o.u.server.util.DmaapSubscriber : response content is
:[]\n","stream":"stdout","time":"2018-06-13T20:52:33.218096153Z"}
{"log":"2018-06-13 20:52:43.401 INFO 451 --- [ main]
o.o.u.server.util.DmaapSubscriber : response content is
:[]\n","stream":"stdout","time":"2018-06-13T20:52:43.402126811Z"}
{"log":"2018-06-13 20:52:53.549 INFO 451 --- [ main]
o.o.u.server.util.DmaapSubscriber : response content is
:[]\n","stream":"stdout","time":"2018-06-13T20:52:53.549390405Z"}
{"log":"2018-06-13 20:53:03.715 INFO 451 --- [ main]
o.o.u.server.util.DmaapSubscriber : response content is
:[]\n","stream":"stdout","time":"2018-06-13T20:53:03.715633889Z"}
{"log":"2018-06-13 20:53:13.866 INFO 451 --- [ main]
o.o.u.server.util.DmaapSubscriber : response content is
:[]\n","stream":"stdout","time":"2018-06-13T20:53:13.867127823Z"}
Sent from Samsung Mobile.
-------- Original message --------
From: Mandeep Khinda
Date:13/06/2018 3:54 PM (GMT+02:00)
To: SEAUDI Abdelmuhaimen OBS/CSO , onap-discuss@lists.onap.org
Subject: Re: [onap-discuss] How to clear logs
Hi,
It is a good question as I have had my disk filled before too.
I have removed the massive .json log file from /var/lib/docker/containers/*/*
in the past after stopping the container and docker daemon but I would not
suggest unless you were okay with having to re-deploy the pod as I have had
mixed success.
I think we need to add some documentation to the K8s cluster build requirements
to enable some sort of log rotation for the docker containers.
I found this:
https://kubernetes.io/docs/concepts/cluster-administration/logging/
Kubernetes currently is not responsible for rotating logs, but rather a
deployment tool should set up a solution to address that. For example, in
Kubernetes clusters, deployed by the kube-up.sh script, there is a
logrotate<https://linux.die.net/man/8/logrotate> tool configured to run each
hour
By default, the docker daemon uses json-file as its logging driver and this
thing comes out of the box with an unlimited size.
https://docs.docker.com/config/containers/logging/json-file/#usage
Try setting “--log-opt max-size=[SIZE]m” in your k8s nodes where docker is
running to prevent this from happening again.
Thanks,
Mandeep
From: <onap-discuss-boun...@lists.onap.org> on behalf of
"abdelmuhaimen.sea...@orange.com" <abdelmuhaimen.sea...@orange.com>
Date: Wednesday, June 13, 2018 at 2:09 AM
To: "onap-discuss@lists.onap.org" <onap-discuss@lists.onap.org>
Subject: [onap-discuss] How to clear logs
Hi,
I was able to resolve an issue with an ONAP OOM Pod that resulted in a log file
filling up 95% of the disk space.
Now, what if I need to clear the logs of an ONAP OOM pod ? What is the correct
process ?
Thanks.
Abdelmuhaimen Seaudi
Orange Labs Egypt
Email: abdelmuhaimen.sea...@orange.com<mailto:abdelmuhaimen.sea...@orange.com>
Mobile: +2012 84644 733
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
_______________________________________________
onap-discuss mailing list
onap-discuss@lists.onap.org
https://lists.onap.org/mailman/listinfo/onap-discuss
