Hello Ottomata,
I'd like you to do a code review. Please visit
https://gerrit.wikimedia.org/r/184223
to review the following change.
Change subject: Sync Hive generated TSVs to stat1002
......................................................................
Sync Hive generated TSVs to stat1002
While the TSVs are already available through the HDFS mount, we want
them on local disks too to avoid unnecessary load on cluster and
network.
Change-Id: Ib8e8bdc4ecb5af138febe2cc24194226b8d7ac9d
---
M manifests/misc/statistics.pp
1 file changed, 8 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/23/184223/1
diff --git a/manifests/misc/statistics.pp b/manifests/misc/statistics.pp
index c03415a..aeb726a 100644
--- a/manifests/misc/statistics.pp
+++ b/manifests/misc/statistics.pp
@@ -516,6 +516,7 @@
# files will be rsynced into /a/log.
"${working_path}/log",
"${working_path}/log/webrequest",
+ "${working_path}/log/webrequest/archive",
]:
ensure => directory,
owner => 'stats',
@@ -558,6 +559,13 @@
source =>
'oxygen.wikimedia.org::udp2log/webrequest/archive/mobile*.gz',
destination => "${working_path}/squid/archive/mobile",
}
+
+ # all webrequest archive logs from hdfs
+ misc::statistics::rsync_job { 'hdfs_webrequest_archive':
+ source => 'stat1002.eqiad.wmnet::hdfs-archive/webrequest/*',
+ destination => "${working_path}/log/webrequest/archive",
+ retention_days => 90, # Pruning after 90 days as those logs contain
private data.
+ }
}
# Class: misc::statistics::rsync::jobs::eventlogging
--
To view, visit https://gerrit.wikimedia.org/r/184223
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib8e8bdc4ecb5af138febe2cc24194226b8d7ac9d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: QChris <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
