Nschaaf has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/335437 )
Change subject: (in progress) Drop wdqs_extract partitions older than 90 days
......................................................................
(in progress) Drop wdqs_extract partitions older than 90 days
TODO: the script that's invoked needs tested
Bug: T146915
Change-Id: I5d29490c0e8e0314a131d12a758bdfb3d2d8735f
---
M modules/role/manifests/analytics_cluster/refinery/data/drop.pp
1 file changed, 11 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/37/335437/1
diff --git a/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
b/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
index fcd9968..91efb1c 100644
--- a/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
+++ b/modules/role/manifests/analytics_cluster/refinery/data/drop.pp
@@ -7,6 +7,7 @@
$webrequest_log_file =
"${role::analytics_cluster::refinery::log_dir}/drop-webrequest-partitions.log"
$eventlogging_log_file =
"${role::analytics_cluster::refinery::log_dir}/drop-eventlogging-partitions.log"
+ $wdqs_extract_log_file =
"${role::analytics_cluster::refinery::log_dir}/drop-wdqs-extract-partitions.log"
# keep this many days of raw webrequest data
$raw_retention_days = 31
@@ -37,4 +38,13 @@
minute => '15',
hour => '*/4',
}
-}
\ No newline at end of file
+
+ # keep this many days of wdqs_extract data
+ $wdqs_extract_retention_days = 90
+ cron {'refinery-drop-wdqs-extract-partitions':
+ command => "export
PYTHONPATH=\${PYTHONPATH}:${role::analytics_cluster::refinery::path}/python &&
${role::analytics_cluster::refinery::path}/bin/refinery-drop-wdqs-extract-partitions
-d ${wdqs_extract_retention_days} -D wmf -l /wmf/data/wmf/wdqs_extract >>
${wdqs_extract_log_file} 2>&1",
+ user => 'hdfs',
+ minute => '15',
+ hour => '*/4',
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/335437
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5d29490c0e8e0314a131d12a758bdfb3d2d8735f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Nschaaf <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
