Ottomata has submitted this change and it was merged.
Change subject: Sync Hive generated TSVs to stat1002
......................................................................
Sync Hive generated TSVs to stat1002
While the TSVs are already available through the HDFS mount, we want
them on local disks too to avoid unnecessary load on cluster and
network.
Change-Id: Ib8e8bdc4ecb5af138febe2cc24194226b8d7ac9d
---
M manifests/misc/statistics.pp
1 file changed, 8 insertions(+), 0 deletions(-)
Approvals:
Ottomata: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/misc/statistics.pp b/manifests/misc/statistics.pp
index c03415a..aeb726a 100644
--- a/manifests/misc/statistics.pp
+++ b/manifests/misc/statistics.pp
@@ -516,6 +516,7 @@
# files will be rsynced into /a/log.
"${working_path}/log",
"${working_path}/log/webrequest",
+ "${working_path}/log/webrequest/archive",
]:
ensure => directory,
owner => 'stats',
@@ -558,6 +559,13 @@
source =>
'oxygen.wikimedia.org::udp2log/webrequest/archive/mobile*.gz',
destination => "${working_path}/squid/archive/mobile",
}
+
+ # all webrequest archive logs from hdfs
+ misc::statistics::rsync_job { 'hdfs_webrequest_archive':
+ source => 'stat1002.eqiad.wmnet::hdfs-archive/webrequest/*',
+ destination => "${working_path}/log/webrequest/archive",
+ retention_days => 90, # Pruning after 90 days as those logs contain
private data.
+ }
}
# Class: misc::statistics::rsync::jobs::eventlogging
--
To view, visit https://gerrit.wikimedia.org/r/184223
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib8e8bdc4ecb5af138febe2cc24194226b8d7ac9d
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: QChris <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
Gerrit-Reviewer: QChris <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
