[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #15 from Antoine hashar Musso (WMF) has...@free.fr --- (In reply to Yuvi Panda from comment #14) Can someone with projectadmin on integration project edit https://wikitech.wikimedia.org/wiki/Hiera:Integration to add the line:

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 Antoine hashar Musso (WMF) has...@free.fr changed: What|Removed |Added Status|PATCH_TO_REVIEW |RESOLVED

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #11 from Greg Grossmeier g...@wikimedia.org --- Just keeping the heat on this bug, we had an outage this morning (times in Eastern US): 07:49 icinga-wm PROBLEM - BetaLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL:

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #12 from Antoine hashar Musso (WMF) has...@free.fr --- I know about two reasons for the HHVM application servers on beta cluster fill /var/ : Bug 73262 - hhvm apache fills /var/log/apache2 with access logs They need to send their

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #13 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 174132 merged by Yuvipanda: puppetmaster: Make time to keep old reports for configurable https://gerrit.wikimedia.org/r/174132 -- You are receiving this mail

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #14 from Yuvi Panda yuvipa...@gmail.com --- Can someone with projectadmin on integration project edit https://wikitech.wikimedia.org/wiki/Hiera:Integration to add the line: puppetmaster::scripts::keep_report_minutes: 360 This will

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #2 from Marc A. Pelletier m...@uberbox.org --- (In reply to Antoine hashar Musso (WMF) from comment #1) /var is /dev/vda2 , I am wondering whether it can be extended somehow. CCing Andrew B and Marc-André. The latest images,

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #3 from Antoine hashar Musso (WMF) has...@free.fr --- I looked at the state of the beta cluster puppet master (deployment-salt). There, /var/lib is a symlink to /srv/var-lib/ which gives more free space. The puppet.master has the

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #4 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 174132 had a related patch set uploaded by Yuvipanda: puppetmaster: Make time to keep old reports for configurable https://gerrit.wikimedia.org/r/174132 -- You

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 Gerrit Notification Bot gerritad...@wikimedia.org changed: What|Removed |Added Status|NEW

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #5 from Antoine hashar Musso (WMF) has...@free.fr --- (In reply to Antoine hashar Musso (WMF) from comment #3) I looked at the state of the beta cluster puppet master (deployment-salt). There, /var/lib is a symlink to

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #6 from Greg Grossmeier g...@wikimedia.org --- (In reply to Antoine hashar Musso (WMF) from comment #1) The /var on labs is indeed only 2GB. puppetmaster reports takes 600MB of disk right now. Can we not just increase the size

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #7 from Andrew Bogott abog...@wikimedia.org --- Greg -- For new instances /var/log is somewhat resizeable. For existing instances you can remount /var/log but that's very messy since every service expects to already have an open

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #8 from Greg Grossmeier g...@wikimedia.org --- (In reply to Andrew Bogott from comment #7) Greg -- For new instances /var/log is somewhat resizeable. How much? Can we just change the default for new deployment-prep instances to

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #9 from Andrew Bogott abog...@wikimedia.org --- (In reply to Greg Grossmeier from comment #8) (In reply to Andrew Bogott from comment #7) Greg -- For new instances /var/log is somewhat resizeable. How much? Can we just

[Bug 73472] [OPS] Jenkins: puppet master fills /var on labs with yaml reports

https://bugzilla.wikimedia.org/show_bug.cgi?id=73472 --- Comment #10 from Greg Grossmeier g...@wikimedia.org --- (In reply to Andrew Bogott from comment #9) (In reply to Greg Grossmeier from comment #8) (In reply to Andrew Bogott from comment #7) Greg -- For new instances /var/log is