Ottomata has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231785
Change subject: Split webrequest camus import job into multiple jobs for
different size topics
......................................................................
Split webrequest camus import job into multiple jobs for different size topics
Change-Id: Ideb4ed885360e9562734526dca1f524506280de8
---
M manifests/role/analytics/refinery.pp
1 file changed, 22 insertions(+), 4 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/85/231785/1
diff --git a/manifests/role/analytics/refinery.pp
b/manifests/role/analytics/refinery.pp
index b449a39..8f4f5e7 100644
--- a/manifests/role/analytics/refinery.pp
+++ b/manifests/role/analytics/refinery.pp
@@ -54,13 +54,31 @@
class role::analytics::refinery::camus {
require role::analytics::refinery
- $camus_webrequest_properties =
"${::role::analytics::refinery::path}/camus/camus.webrequest.properties"
- $camus_webrequest_log_file =
"${::role::analytics::refinery::log_dir}/camus-webrequest.log"
- cron { 'refinery-camus-webrequest-import':
- command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest-import ${camus_webrequest_properties} >>
${camus_webrequest_log_file} 2>&1",
+ # TODO: Move these Camus .properties files to puppet erb templates.
+
+ # Split up high and low volume webrequest Camus imports into multiple jobs.
+ # This will help keep large volume imports from causing lower volume ones
+ # to lag.
+
+ # webrequest_text
+ cron { 'refinery-camus-webrequest_text-import':
+ command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest_text-import
${::role::analytics::refinery::path}/camus/camus.webrequest_text.properties >>
${::role::analytics::refinery::log_dir}/camus-webrequest_text.log 2>&1",
user => 'hdfs', # we might want to use a different user for this,
not sure.
minute => '*/10',
}
+ # webrequest_upload
+ cron { 'refinery-camus-webrequest_upload-import':
+ command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest_upload-import
${::role::analytics::refinery::path}/camus/camus.webrequest_upload.properties
>> ${::role::analytics::refinery::log_dir}/camus-webrequest_upload.log 2>&1",
+ user => 'hdfs', # we might want to use a different user for this,
not sure.
+ minute => '*/10',
+ }
+ # webrequest_text
+ cron { 'refinery-camus-webrequest-small-import':
+ command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest-small-import
${::role::analytics::refinery::path}/camus/camus.webrequest-small.properties >>
${::role::analytics::refinery::log_dir}/camus-webrequest-small.log 2>&1",
+ user => 'hdfs', # we might want to use a different user for this,
not sure.
+ minute => '*/10',
+ }
+
$camus_eventlogging_properties =
"${::role::analytics::refinery::path}/camus/camus.eventlogging.properties"
$camus_eventlogging_log_file =
"${::role::analytics::refinery::log_dir}/camus-eventlogging.log"
--
To view, visit https://gerrit.wikimedia.org/r/231785
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ideb4ed885360e9562734526dca1f524506280de8
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
