[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[rhtyd]

Github user rhtyd commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
LGTM @karuturi this is ready for merge
Ping @abhinandanprateek in case you want to review?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
LGTM.  tag:mergeready


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@serbaut Can you do a force push to kick off jenkins again.  I'm guessing 
Jenkins just had an issue and not the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[serbaut]

Github user serbaut commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
Ok. I removed it from rsyslog since it should be safe there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[leprechau]

Github user leprechau commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
We always want `compress` ... but the only time you need or want 
`delaycompress` is if you can't be sure that the program writing to the log can 
be successfully told to stop appending to that log.  In the case where you are 
certain that the writing program is going to do the right thing there is no 
need to add `delaycompress` as it just takes up extra space in already rotated 
logs until the next iteration.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[serbaut]

Github user serbaut commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
Is it safe to remove delaycompress across the board, I assume it is there 
for a reason?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@serbaut I agree with @ustcweizhou.  Please remove delaycompress and up to 
10.  I'd like to get this PR in as it is the second part of the problem 
resolution for my issue.  After that LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[kiwiflyer]

Github user kiwiflyer commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
We pulled this in and tested it alongside 
https://github.com/apache/cloudstack/pull/1954


![image](https://cloud.githubusercontent.com/assets/17278194/23148479/b8c73e42-f7ab-11e6-8ffc-b884f802b520.png)

We did remove the delay compress in our testing.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[ustcweizhou]

Github user ustcweizhou commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@serbaut maxsize is fine for me.
Could you please remove delaycompress, and increase rotate in 
systemvm/patches/debian/config/etc/logrotate.d/cloud (eg from 4 to 10) ?
LTGM after that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[serbaut]

Github user serbaut commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@dmabry yes we are using redundant VPC routers. 

Using `size` instead of `maxsize` would delay the rotation of logs until 
they reach `size`. 10 logfiles with size 10MB could consume 200MB due to 
deplaycompress. Using `maxsize` would rotate the slowly growing logfiles daily 
and avoid this issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@ustcweizhou Thanks for taking the time to test as well.  I confirm in our 
testing over the weekend that we were able to consistently 
create/destroy/expunge VMs every 2 minutes without issue.  Before the PR 
https://github.com/apache/cloudstack/pull/1954 and this PR, we were having the 
VPC VR fall over ~8 hours into the test.

In summary, this PR with the one I submitted this morning solved our issues.

Thanks again for the help.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
I have submitted a PR against master that fixes cloud-early-config.
https://github.com/apache/cloudstack/pull/1954



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[ustcweizhou]

Github user ustcweizhou commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
1. As @dmabry said, the logrotate is moved from cron.daily to cron.hourly 
in isolated network VR in commit.
We should make similar change for VPC VR and VR for shared network.
```
commit 6ac06e5e5e3ceed4a3e3a86ea5f82ffb59c266f2
Author: Abhinandan Prateek 
AuthorDate: Mon Mar 30 11:36:11 2015 +0530
Commit: Abhinandan Prateek 
CommitDate: Mon Mar 30 13:49:23 2015 +0530

CLOUDSTACK-6885: rotate logs hourly on VR
---
 .../debian/config/etc/init.d/cloud-early-config|4 
 systemvm/patches/debian/config/etc/logrotate.conf  |2 +-
 .../patches/debian/config/etc/logrotate.d/cloud|3 ---
 .../patches/debian/config/etc/logrotate.d/rsyslog  |2 ++
 4 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/systemvm/patches/debian/config/etc/init.d/cloud-early-config 
b/systemvm/patches/debian/config/etc/init.d/cloud-early-config
index 729b58f..5a2ee6f 100755
--- a/systemvm/patches/debian/config/etc/init.d/cloud-early-config
+++ b/systemvm/patches/debian/config/etc/init.d/cloud-early-config
@@ -995,6 +995,10 @@ setup_router() {
   sed -i "s/-A INPUT -i eth0 -p udp -m udp --dport 53 -j ACCEPT/-A INPUT 
-i eth0 -p udp -m udp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" 
/etc/iptables/rules
   sed -i "s/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT/-A INPUT 
-i eth0 -p tcp -m tcp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" 
/etc/iptables/rules.v4
   sed -i "s/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT/-A INPUT 
-i eth0 -p tcp -m tcp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" 
/etc/iptables/rules
+
+  #setup hourly logrotate
+  mv -n /etc/cron.daily/logrotate /etc/cron.hourly 2>&1
+
 }
```

2. size and maxsize have no difference in my testing. Anyway, maxsize is ok 
for me.

3. As I said in previous comment, we'd better use more 'rotate' for 
cloud.log

4. @dmabry If you are not confident with hourly file compress, we can add a 
file in /etc/cron.d/
```
# cat /etc/cron.d/logrorate.cloud
* * * * * root /usr/sbin/logrotate /etc/logrotate.d/cloud 2>&1 > /dev/null
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@serbaut Just curious, are you using VPC routers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[ustcweizhou]

Github user ustcweizhou commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@dmabry I will test it and update you.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@ustcweizhou - Thanks for the +1 on cron.hourly move.  From looking at the 
code, it looks like non-redundant VRs are already moving to cron.hourly, but 
vpcrouter isn't.  In our case, we are using vpcrouter.  I'm working on a PR 
against master that will fix the vpcrouter flavor of this code.  I'll link the 
PR here when I'm done testing.

I'd also like to understand why you would prefer 'size' over 'maxsize'.  I 
know the difference is subtle and it really won't apply to anything that is set 
to "hourly", but it could impact the dailys.  In the VR case with a limited 
/var/log partition size, I would think that the 'maxsize' option would be 
preferable.  It would keep run away log files in check by rotating them even 
before their interval is reached.

```size size
  Log files are rotated only if they grow bigger then size 
bytes.
  If size is followed by k, the size is assumed to  be  in  
kilo-
  bytes.  If the M is used, the size is in megabytes, and if G 
is
  used, the size is in gigabytes. So size 100,  size  100k,  
size
  100M and size 100G are all valid.
maxsize size
  Log files are rotated when they grow bigger than size bytes 
even before
  the additionally specified time interval (daily, weekly, 
monthly, 
  or yearly).  The related size option is  similar  except  
that  it 
  is mutually exclusive with the time interval options, and it 
causes
  log files to be rotated without regard for the last rotation 
time.  
  When maxsize is used, both the size and timestamp of a log 
file are  
  considered.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[ustcweizhou]

Github user ustcweizhou commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
agree with @dmabry 
logrotate need to be moved (or copied) to cron.hourly
```
diff --git a/systemvm/patches/debian/config/etc/cron.hourly/logrotate 
b/systemvm/patches/debian/config/etc/cron.hourly/logrotate
new file mode 100755
index 000..055b4d5
--- /dev/null
+++ b/systemvm/patches/debian/config/etc/cron.hourly/logrotate
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+test -x /usr/sbin/logrotate || exit 0
+/usr/sbin/logrotate /etc/logrotate.conf
+
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[dmabry]

Github user dmabry commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
Agreed with @leprechau .  I don't think just changing logrotate to compress 
and rotate at smaller size intervals will not completely solve our issues here. 
 We really need to adjust logrotate to run every hour, or make cloud.log less 
verbose by default.  I'm looking into seeing if it would be viable to make 
logrotate run every hour, but I believe that will require changing how the 
systemvm image is built by moving logrotate from cron.daily to cron.hourly.  
Not a huge deal, but I think it might be right approach over all since we have 
limited disk space in /var/log.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[leprechau]

Github user leprechau commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
We are seeing issues under heavy user interaction cycles with redundant 
VRs.  In testing you can reliably completely fill the `/var/log/` partition 
with data primarily from `cloud.log` when you create/stop/delete/expunge ~200 
VMs over a 24 hour period with the default settings.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...

[ustcweizhou]

Github user ustcweizhou commented on the issue:

https://github.com/apache/cloudstack/pull/1915
  
@serbaut I agree with reducing the size.

However, I suggest to 
(1) remove the delaycompress, then the files like syslog.1 or rsyslog.1 
will be compressed.
(2) keep rotate , as the compressed file are quite small (< 1MB). Keeping 
more compressing files (especially cloud.log) will be helpful for 
troubleshooting in case we have some issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---