[
https://issues.apache.org/jira/browse/CLOUDSTACK-9155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15054944#comment-15054944
]
ASF GitHub Bot commented on CLOUDSTACK-9155:
--------------------------------------------
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164255438
The new script ends up on the router:
```
root@r-15-VM:~# cat /etc/logrotate.d/cloud
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
/var/log/cloud.log {
rotate 4
daily
size 10M
missingok
notifempty
compress
delaycompress
# CLOUDSTACK-9155: We cannot tell the processes that are writing to
this
# file to use the new inode, so instead we copy the original file,
truncate
# it and keep the same inode.
copytruncate
}
```
Truncated list of processes that have `cloud.log` open:
```
apache2 3636 3686 www-data 3w REG 254,10 264891
59 /var/log/cloud.log
apache2 3636 3687 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
apache2 3636 3688 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
apache2 3636 3689 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
apache2 3636 3690 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
conntrack 3837 root 3w REG 254,10 265499
59 /var/log/cloud.log
keepalive 3851 root 3w REG 254,10 265499
59 /var/log/cloud.log
keepalive 3852 root 3w REG 254,10 265499
59 /var/log/cloud.log
keepalive 3853 root 3w REG 254,10 265499
59 /var/log/cloud.log
_plutorun 4141 root 3w REG 254,10 265499
59 /var/log/cloud.log
logger 4142 root 3w REG 254,10 265499
59 /var/log/cloud.log
_plutorun 4145 root 3w REG 254,10 265499
59 /var/log/cloud.log
_plutoloa 4146 root 3w REG 254,10 265499
59 /var/log/cloud.log
xl2tpd 4203 root 3w REG 254,10 265499
59 /var/log/cloud.log
passwd_se 4225 root 1w REG 254,10 265499
59 /var/log/cloud.log
passwd_se 4225 root 2w REG 254,10 265499
59 /var/log/cloud.log
passwd_se 4225 root 3w REG 254,10 265499
59 /var/log/cloud.log
python 4229 root 3w REG 254,10 265499
59 /var/log/cloud.log
update_co 4927 root 3r REG 254,10 265499
59 /var/log/cloud.log
```
Force logrotate:
```
logrotate /etc/logrotate.conf -f
```
Same processes, they still have `cloud.log` open (instead of `cloud.log.1`)
```
apache2 3636 3686 www-data 3w REG 254,10 264891
59 /var/log/cloud.log
apache2 3636 3687 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
apache2 3636 3688 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
apache2 3636 3689 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
apache2 3636 3690 www-data 3w REG 254,10 265499
59 /var/log/cloud.log
conntrack 3837 root 3w REG 254,10 265499
59 /var/log/cloud.log
keepalive 3851 root 3w REG 254,10 265499
59 /var/log/cloud.log
keepalive 3852 root 3w REG 254,10 265499
59 /var/log/cloud.log
keepalive 3853 root 3w REG 254,10 265499
59 /var/log/cloud.log
_plutorun 4141 root 3w REG 254,10 265499
59 /var/log/cloud.log
logger 4142 root 3w REG 254,10 265499
59 /var/log/cloud.log
_plutorun 4145 root 3w REG 254,10 265499
59 /var/log/cloud.log
_plutoloa 4146 root 3w REG 254,10 265499
59 /var/log/cloud.log
xl2tpd 4203 root 3w REG 254,10 265499
59 /var/log/cloud.log
passwd_se 4225 root 1w REG 254,10 265499
59 /var/log/cloud.log
passwd_se 4225 root 2w REG 254,10 265499
59 /var/log/cloud.log
passwd_se 4225 root 3w REG 254,10 265499
59 /var/log/cloud.log
python 4229 root 3w REG 254,10 265499
59 /var/log/cloud.log
update_co 4927 root 3r REG 254,10 265499
59 /var/log/cloud.log
```
Deploying this to our clouds running 4.7.
> Log rotate of cloud.log doesn't work properly
> ---------------------------------------------
>
> Key: CLOUDSTACK-9155
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9155
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Virtual Router
> Affects Versions: 4.6.0, 4.7.0, 4.6.1
> Reporter: Remi Bergsma
> Assignee: Remi Bergsma
> Priority: Critical
> Fix For: 4.7.0, 4.6.2
>
>
> Many processes log into the cloud.log file. When log rotate is called, many
> of them keep logging to the old inode and fill up the disk like that.
> These have cloud.log open:
> ```
> root@r-1023-VM:~# lsof| grep cloud | awk {'print $1'} | sort -u
> apache2
> conntrack
> keepalive
> logger
> passwd_se
> _plutoloa
> _plutorun
> python
> xl2tpd
> ```
> Current log rotate config:
> ```
> /var/log/cloud.log {
> rotate 4
> daily
> size 10M
> missingok
> notifempty
> compress
> delaycompress
> }
> ```
> After log rotate this happens:
> ```
> root@r-996-VM:/etc# lsof | grep cloud.log.1
> _plutorun 767 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> logger 768 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> _plutorun 772 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> _plutoloa 773 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> xl2tpd 843 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> python 854 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> passwd_se 860 root 1w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> passwd_se 860 root 2w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> passwd_se 860 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> python 863 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> passwd_se 869 root 1w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> passwd_se 869 root 2w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> passwd_se 869 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> python 871 root 3w REG 202,10 26054919
> 71 /var/log/cloud.log.1
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
