Ottomata has submitted this change and it was merged. Change subject: Set appropriate # mappers for new camus jobs ......................................................................
Set appropriate # mappers for new camus jobs Change-Id: I166302b2674ec1b330e2e71955f6bb9008137a5f --- M camus/README.md M camus/camus.webrequest-small.properties M camus/camus.webrequest_text.properties M camus/camus.webrequest_upload.properties 4 files changed, 8 insertions(+), 11 deletions(-) Approvals: Ottomata: Verified; Looks good to me, approved diff --git a/camus/README.md b/camus/README.md index 9feb2e0..8ef6614 100644 --- a/camus/README.md +++ b/camus/README.md @@ -3,12 +3,11 @@ # webrequest_* The webrequest_maps webrequest_mobile and webrequest_misc topics are relatively -small volume. This camus job is expected to complete runs fairly quickly. +small volume. The webrequest-small camus job is expected to complete runs fairly quickly. webrequest_text and webrequest_upload are both large volume. These are run as separate jobs so as not to interere with each other and smaller volume webrequest imports. - The camus.webrequest.properties file contains all webrequest topics, but as of 2015-08-15 is deprecated. \ No newline at end of file diff --git a/camus/camus.webrequest-small.properties b/camus/camus.webrequest-small.properties index 69e9ff2..1f33aca 100644 --- a/camus/camus.webrequest-small.properties +++ b/camus/camus.webrequest-small.properties @@ -41,9 +41,9 @@ etl.record.writer.provider.class=com.linkedin.camus.etl.kafka.common.SequenceFileRecordWriterProvider # Max hadoop tasks to use, each task can pull multiple topic partitions. -# Currently 5 topics, each with 12 partitions. Setting -# map.tasks to 5*12. -mapred.map.tasks=60 +# Currently 3 topics, each with 12 partitions. Setting +# map.tasks to 3*12. +mapred.map.tasks=36 # Connection parameters. kafka.brokers=analytics1012.eqiad.wmnet:9092,analytics1018.eqiad.wmnet:9092,analytics1021.eqiad.wmnet:9092,analytics1022.eqiad.wmnet:9092 diff --git a/camus/camus.webrequest_text.properties b/camus/camus.webrequest_text.properties index 19154e9..a192fdd 100644 --- a/camus/camus.webrequest_text.properties +++ b/camus/camus.webrequest_text.properties @@ -41,9 +41,8 @@ etl.record.writer.provider.class=com.linkedin.camus.etl.kafka.common.SequenceFileRecordWriterProvider # Max hadoop tasks to use, each task can pull multiple topic partitions. -# Currently 5 topics, each with 12 partitions. Setting -# map.tasks to 5*12. -mapred.map.tasks=60 +# Currently 1 topics, each with 12 partitions. +mapred.map.tasks=12 # Connection parameters. kafka.brokers=analytics1012.eqiad.wmnet:9092,analytics1018.eqiad.wmnet:9092,analytics1021.eqiad.wmnet:9092,analytics1022.eqiad.wmnet:9092 diff --git a/camus/camus.webrequest_upload.properties b/camus/camus.webrequest_upload.properties index c0aa006..9e503ab 100644 --- a/camus/camus.webrequest_upload.properties +++ b/camus/camus.webrequest_upload.properties @@ -41,9 +41,8 @@ etl.record.writer.provider.class=com.linkedin.camus.etl.kafka.common.SequenceFileRecordWriterProvider # Max hadoop tasks to use, each task can pull multiple topic partitions. -# Currently 5 topics, each with 12 partitions. Setting -# map.tasks to 5*12. -mapred.map.tasks=60 +# Currently 1 topics, each with 12 partitions. +mapred.map.tasks=12 # Connection parameters. kafka.brokers=analytics1012.eqiad.wmnet:9092,analytics1018.eqiad.wmnet:9092,analytics1021.eqiad.wmnet:9092,analytics1022.eqiad.wmnet:9092 -- To view, visit https://gerrit.wikimedia.org/r/231784 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I166302b2674ec1b330e2e71955f6bb9008137a5f Gerrit-PatchSet: 1 Gerrit-Project: analytics/refinery Gerrit-Branch: master Gerrit-Owner: Ottomata <[email protected]> Gerrit-Reviewer: Ottomata <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
