Hi, After 4 days, the installation is still stable, however, still the var/log/syslog and /var/log/syslog.1 have reached 44 GB space again.
-rw-r----- 1 syslog adm 7043982261 Jun 4 10:23 syslog -rw-r----- 1 syslog adm 37589664321 Jun 4 06:25 syslog.1 -rw-r----- 1 syslog adm 99405 Jun 3 06:25 syslog.2.gz -rw-r----- 1 syslog adm 79388 Jun 2 06:25 syslog.3.gz -rw-r----- 1 syslog adm 155344 Jun 1 06:25 syslog.4.gz Abdelmuhaimen Seaudi Orange Labs Egypt Email: abdelmuhaimen.sea...@orange.com<mailto:abdelmuhaimen.sea...@orange.com> Mobile: +2012 84644 733 From: SEAUDI Abdelmuhaimen OBS/CSO Sent: Saturday, June 2, 2018 1:01 PM To: onap-discuss@lists.onap.org Subject: /var/log/syslog taking up a lot of space on OOM Beijing Hi, I have an OOM Beijing instance running on 1 VM for Rancher Server, and 3 VMs Rancher hosts, each with 8 vCPUs, 52 GB RAM, 50 GB Root Parition, and 100 GB 2nd Partition for /var/lib/docker/. After running for 2 days, the installation is stable so far, only OOM-SNIRO gives FAIL in robot health check, and only pod onap-oof is failing, since the installation. However, I noticed one of the nodes taking a lot of root storage space, and I found out it's /var/log/syslog and /var/log/syslog.1, which are taking ~43 GB of space. What is the reason for this behaviour ? root@olc-bjng-2:~# free -h total used free shared buff/cache available Mem: 51G 25G 12G 557M 13G 24G Swap: 0B 0B 0B root@olc-bjng-2:~# df -h /dev/vda1 /dev/vdb Filesystem Size Used Avail Use% Mounted on /dev/vda1 49G 44G 5.0G 90% / <<<<<<<<<<<<<<<<<<<< a lot of space, not from /var/lib/docker /dev/vdb 99G 30G 64G 32% /mnt root@olc-bjng-2:~# root@olc-bjng-2:/var/log# ls -l total 42153316 ... -rw-r----- 1 syslog adm 9811749726 Jun 2 10:40 syslog -rw-r----- 1 syslog adm 33351210056 Jun 2 06:25 syslog.1 -rw-r----- 1 syslog adm 149784 Jun 1 06:25 syslog.2.gz I see the following lines near the top of syslog.1 Jun 1 08:58:44 olc-bjng-2 dockerd[9856]: time="2018-06-01T08:58:44.599081414Z" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container 6b3b60d6dde71c0ea16b698b1b7a964e53fb3dd54bdf1e198a49d51f117a2c40" Jun 1 08:58:45 olc-bjng-2 dockerd[9856]: time="2018-06-01T08:58:45.907586780Z" level=error msg="Handler for GET /v1.22/containers/952333fb19fec201d9adf226847d8a3e21045d831fe73c10082fe1124271f31a/json returned error: No such container: 952333fb19fec201d9adf226847d8a3e21045d831fe73c10082fe1124271f31a" Jun 1 08:59:43 olc-bjng-2 dockerd[9856]: time="2018-06-01T08:59:43.372667000Z" level=warning msg="failed to close stdin: rpc error: code = 2 desc = write /var/run/docker/libcontainerd/containerd/4780e1142fbea7740b0eded42057be2ab17cc0998b2319596654a9a3606b9045/6f635acd5a26750d0f84ee506ed28c4b18b39a64c8994966636d138a88db8e6c/control: bad file descriptor" Jun 1 09:02:28 olc-bjng-2 dockerd[9856]: time="2018-06-01T09:02:28.564875759Z" level=error msg="Handler for DELETE /v1.27/images/sha256:b7fa6b9cb097d4be9c482f44a2ab2d84d0067b598f80dacfedc11b30feaf2fc6 returned error: conflict: unable to delete b7fa6b9cb097 (cannot be forced) - image is being used by running container c55cddbe095c" Jun 1 09:02:28 olc-bjng-2 dockerd[9856]: time="2018-06-01T09:02:28.568346021Z" level=error msg="Handler for DELETE /v1.27/images/sha256:14de771cc17886ac2b6e6eace825c8c67afb59b88804944421a5c2dbebe1ddaf returned error: conflict: unable to delete 14de771cc178 (cannot be forced) - image is being used by running container 1f3008e7bc83" Jun 1 09:02:28 olc-bjng-2 dockerd[9856]: time="2018-06-01T09:02:28.570397502Z" level=error msg="Handler for DELETE /v1.27/images/sha256:bd33f8c865b1cefab6e876006de8542892e21205f33ecd0de82c778be72a2b39 returned error: conflict: unable to delete bd33f8c865b1 (cannot be forced) - image is being used by running container 82a6677eaf0f" And i see the following lines near the bottom of syslog.1 Jun 2 06:25:01 olc-bjng-2 dockerd[9856]: time="2018-06-02T06:25:01.634823245Z" level=error msg="Failed to log msg \"\\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:419)\" for logger json-file: write /mnt/containers/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a-json.log: no space left on device" Jun 2 06:25:01 olc-bjng-2 dockerd[9856]: time="2018-06-02T06:25:01.635142774Z" level=error msg="Failed to log msg \"\\tat org.glassfish.jersey.client.InboundJaxrsResponse.readEntity(InboundJaxrsResponse.java:108)\" for logger json-file: write /mnt/containers/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a-json.log: no space left on device" Jun 2 06:25:01 olc-bjng-2 dockerd[9856]: time="2018-06-02T06:25:01.635522112Z" level=error msg="Failed to log msg \"\\tat org.onap.usecaseui.server.util.DmaapSubscriber.getDMaaPData(DmaapSubscriber.java:112)\" for logger json-file: write /mnt/containers/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a-json.log: no space left on device" Jun 2 06:25:01 olc-bjng-2 dockerd[9856]: time="2018-06-02T06:25:01.636113243Z" level=error msg="Failed to log msg \"\\tat org.onap.usecaseui.server.util.DmaapSubscriber.subscribe(DmaapSubscriber.java:79)\" for logger json-file: write /mnt/containers/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a-json.log: no space left on device" Jun 2 06:25:01 olc-bjng-2 dockerd[9856]: time="2018-06-02T06:25:01.636429653Z" level=error msg="Failed to log msg \"\\tat org.onap.usecaseui.server.util.DmaapSubscriber.run(DmaapSubscriber.java:136)\" for logger json-file: write /mnt/containers/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a-json.log: no space left on device" Jun 2 06:25:01 olc-bjng-2 dockerd[9856]: time="2018-06-02T06:25:01.636802049Z" level=error msg="Failed to log msg \"\\tat org.onap.usecaseui.server.UsecaseuiServerApplication.main(UsecaseuiServerApplication.java:44)\" for logger json-file: write /mnt/containers/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a/36441e295ace461c6c1e5a2a60f8004272c056619d381846349ce47c8900385a-json.log: no space left on device" _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
_______________________________________________ onap-discuss mailing list onap-discuss@lists.onap.org https://lists.onap.org/mailman/listinfo/onap-discuss
