Tim Landscheidt has uploaded a new change for review.
https://gerrit.wikimedia.org/r/80332
Change subject: WIP: Tools: Add infrastructure for AWStats
......................................................................
WIP: Tools: Add infrastructure for AWStats
WORK IN PROGRESS, DO NOT COMMIT
This change is part of an effort to simplify and standardize the web
server log creation and distribution.
Previously, access logs were collected in a distributed manner on all
webservers and saved on an NFS share. Instead this change siphons the
data centrally from the web proxy.
A significant problem for the processing of the log data by AWStats is
the daily log rotation. Synchronizing this process between different
hosts, perhaps even a grid job, is nontrivial. Instead, we run the
AWStats update process on the web proxy, hourly per cron and on demand
from logrotate. The memory and CPU footprint of the AWStats update
process is small enough that there is no concern that this could
impede the proxy's performance so it does not warrant replacing the
direct execution of the AWStats update process with for example a
spool system.
To redact the log, we use a sed command instead of logsplitter so that
we do not have to worry about managing the latter's compilation and
deployment.
Another challenge is redundancy of data. While we could save the
redacted version of the log to disk, it is not necessary from a time
or load viewpoint compared to dynamic redaction. Also, as this would
likely mean that the redacted log would be in the realm of the awstats
tool but writable by root as the web server process, we would have to
be very careful to not open up security holes. Instead we rely on
dynamic redaction and sudo for privileges.
Change-Id: Ic7c542214c754e0e790308b1fae1aeba1cc2890a
---
A modules/toollabs/files/awstats/awstats-filter.sh
A modules/toollabs/files/awstats/logrotate
A modules/toollabs/files/awstats/sudoers
M modules/toollabs/manifests/webproxy.pp
4 files changed, 56 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/32/80332/1
diff --git a/modules/toollabs/files/awstats/awstats-filter.sh
b/modules/toollabs/files/awstats/awstats-filter.sh
new file mode 100644
index 0000000..38db36a
--- /dev/null
+++ b/modules/toollabs/files/awstats/awstats-filter.sh
@@ -0,0 +1,5 @@
+#!/bin/bash
+
+# This file is managed by Puppet.
+
+exec sed -ne 's/^[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+ - - \(\[[^]]\+\]\)
"\(GET\) \/\([^\/]\+\)\/.* \(HTTP\/1.1\)" \([0-9]\{3\}\) \([0-9]\+\)
.*$/127.0.0.1 - - \1 "\2 \/\3 \4" \5 \6/p;' /var/log/apache2/access.log
diff --git a/modules/toollabs/files/awstats/logrotate
b/modules/toollabs/files/awstats/logrotate
new file mode 100644
index 0000000..f968d18
--- /dev/null
+++ b/modules/toollabs/files/awstats/logrotate
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+# This file is managed by Puppet.
+
+exec /usr/bin/sudo -niu local-awstats /data/project/awstats/refresh-awstats.sh
diff --git a/modules/toollabs/files/awstats/sudoers
b/modules/toollabs/files/awstats/sudoers
new file mode 100644
index 0000000..20f2bb0
--- /dev/null
+++ b/modules/toollabs/files/awstats/sudoers
@@ -0,0 +1,5 @@
+# sudo policy for AWStats.
+
+# This file is managed by Puppet.
+
+local-awstats tools-webproxy = NOPASSWD: /usr/local/sbin/awstats-filter.sh
diff --git a/modules/toollabs/manifests/webproxy.pp
b/modules/toollabs/manifests/webproxy.pp
index eb044f6..4dc08f8 100644
--- a/modules/toollabs/manifests/webproxy.pp
+++ b/modules/toollabs/manifests/webproxy.pp
@@ -14,5 +14,45 @@
include toollabs::infrastructure
#TODO: apache config
-}
+ # AWStats infrastructure.
+ file { "/usr/local/sbin/awstats-filter.sh":
+ ensure => file,
+ mode => "0755",
+ owner => "root",
+ group => "root",
+ source => "puppet:///modules/toollabs/awstats/awstats-filter.sh";
+ }
+
+ file { "/etc/logrotate.d/httpd-prerotate":
+ ensure => directory,
+ mode => "0755",
+ owner => "root",
+ group => "root";
+ }
+
+ file { "/etc/logrotate.d/httpd-prerotate/awstats.sh":
+ ensure => file,
+ mode => "0755",
+ owner => "root",
+ group => "root",
+ require => File["/etc/logrotate.d/httpd-prerotate"],
+ source => "puppet:///modules/toollabs/awstats/logrotate";
+ }
+
+ file { "/etc/sudoers.d/awstats":
+ ensure => file,
+ mode => "0440",
+ owner => "root",
+ group => "root",
+ source => "puppet:///modules/toollabs/awstats/sudoers";
+ }
+
+ cron { "awstats-update":
+ ensure => present,
+ minute => 14,
+ hour => "*",
+ user => local-awstats,
+ command => "/data/project/awstats/refresh-awstats.sh";
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/80332
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic7c542214c754e0e790308b1fae1aeba1cc2890a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Tim Landscheidt <t...@tim-landscheidt.de>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
