Ottomata has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231795
Change subject: Revert split of webrequest imports in multiple Camus jobs
......................................................................
Revert split of webrequest imports in multiple Camus jobs
I think the Camus offset history will get complicated if we don't coordinate
this really well.
Instead, I will just make it so that each Camus job only runs for 9 minutes, so
as not to
starve out other topics.
Change-Id: I4f1a3c6f489e76dbb49b49aaf6674c3605e9b547
---
M manifests/role/analytics/refinery.pp
1 file changed, 5 insertions(+), 20 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/95/231795/1
diff --git a/manifests/role/analytics/refinery.pp
b/manifests/role/analytics/refinery.pp
index 8f4f5e7..956e631 100644
--- a/manifests/role/analytics/refinery.pp
+++ b/manifests/role/analytics/refinery.pp
@@ -54,31 +54,16 @@
class role::analytics::refinery::camus {
require role::analytics::refinery
- # TODO: Move these Camus .properties files to puppet erb templates.
+ # TODO: Move these .properties files to puppet erb templates.
- # Split up high and low volume webrequest Camus imports into multiple jobs.
- # This will help keep large volume imports from causing lower volume ones
- # to lag.
+ $camus_webrequest_properties =
"${::role::analytics::refinery::path}/camus/camus.webrequest.properties"
+ $camus_webrequest_log_file =
"${::role::analytics::refinery::log_dir}/camus-webrequest.log"
- # webrequest_text
- cron { 'refinery-camus-webrequest_text-import':
- command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest_text-import
${::role::analytics::refinery::path}/camus/camus.webrequest_text.properties >>
${::role::analytics::refinery::log_dir}/camus-webrequest_text.log 2>&1",
+ cron { 'refinery-camus-webrequest-import':
+ command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest-import ${camus_webrequest_properties} >>
${camus_webrequest_log_file} 2>&1",
user => 'hdfs', # we might want to use a different user for this,
not sure.
minute => '*/10',
}
- # webrequest_upload
- cron { 'refinery-camus-webrequest_upload-import':
- command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest_upload-import
${::role::analytics::refinery::path}/camus/camus.webrequest_upload.properties
>> ${::role::analytics::refinery::log_dir}/camus-webrequest_upload.log 2>&1",
- user => 'hdfs', # we might want to use a different user for this,
not sure.
- minute => '*/10',
- }
- # webrequest_text
- cron { 'refinery-camus-webrequest-small-import':
- command => "${::role::analytics::refinery::path}/bin/camus --job-name
refinery-camus-webrequest-small-import
${::role::analytics::refinery::path}/camus/camus.webrequest-small.properties >>
${::role::analytics::refinery::log_dir}/camus-webrequest-small.log 2>&1",
- user => 'hdfs', # we might want to use a different user for this,
not sure.
- minute => '*/10',
- }
-
$camus_eventlogging_properties =
"${::role::analytics::refinery::path}/camus/camus.eventlogging.properties"
$camus_eventlogging_log_file =
"${::role::analytics::refinery::log_dir}/camus-eventlogging.log"
--
To view, visit https://gerrit.wikimedia.org/r/231795
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4f1a3c6f489e76dbb49b49aaf6674c3605e9b547
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
