ArielGlenn has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/386388 )
Change subject: generate one config file for xml/sql dumps for wikis
......................................................................
generate one config file for xml/sql dumps for wikis
We had separate files for: big wikis, enwiki, wikidatawiki, the rest,
media, monitor service.
This commit combines them all into one. Settings specific to
enwiki are collected in one section, same for wikidatawiki, big wikis,
media dumps, and so on.
All cron jobs are converted over to use the new override sections
to get the settings they need, where needed.
This will help facilitate the move of xml/sql dumps to the dumpsdata
hosts.
Bug: T178893
Change-Id: I6fc98e3152693b073591da1c582aa9cb9c9b1411
---
M modules/snapshot/files/cron/create-media-per-project-lists.sh
M modules/snapshot/files/cron/list-last-good-dumps.sh
M modules/snapshot/manifests/cron/pagetitles.pp
M modules/snapshot/manifests/dumps/configs.pp
M modules/snapshot/manifests/dumps/monitor.pp
M modules/snapshot/manifests/dumps/stagesconfig.pp
M modules/snapshot/manifests/dumps/wikiconf.pp
M modules/snapshot/templates/dumps/fulldumps.sh.erb
M modules/snapshot/templates/dumps/wikidump.conf.erb
M modules/snapshot/templates/initscripts/dumps-monitor.systemd.erb
M modules/snapshot/templates/initscripts/dumps-monitor.upstart.erb
11 files changed, 225 insertions(+), 207 deletions(-)
Approvals:
ArielGlenn: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/snapshot/files/cron/create-media-per-project-lists.sh
b/modules/snapshot/files/cron/create-media-per-project-lists.sh
index c9342c1..9e8f7fe 100755
--- a/modules/snapshot/files/cron/create-media-per-project-lists.sh
+++ b/modules/snapshot/files/cron/create-media-per-project-lists.sh
@@ -9,7 +9,7 @@
DATE=`/bin/date '+%Y%m%d'`
outputdir="${datadir}/public/other/imageinfo/$DATE"
-configfile="${confsdir}/wikidump.conf.media"
+configfile="${confsdir}/wikidump.conf:media"
errors=0
cd "$repodir"
diff --git a/modules/snapshot/files/cron/list-last-good-dumps.sh
b/modules/snapshot/files/cron/list-last-good-dumps.sh
index 84d8ee0..d00d3e7 100644
--- a/modules/snapshot/files/cron/list-last-good-dumps.sh
+++ b/modules/snapshot/files/cron/list-last-good-dumps.sh
@@ -7,7 +7,7 @@
source /usr/local/etc/set_dump_dirs.sh
# generate lists of most recent completed successful dumps for rsync (dirs,
files)
-python /usr/local/bin/list-last-n-good-dumps.py --dumpsnumber 1,2,3,4,5
--dirlisting 'rsync-dirlist-last-%s-good.txt' --configfile
"${confsdir}/wikidump.conf.monitor" --rsynclists --relpath --outputdir
"${datadir}/public/"
-python /usr/local/bin/list-last-n-good-dumps.py --dumpsnumber 1,2,3,4,5
--filelisting 'rsync-filelist-last-%s-good.txt' --configfile
"${confsdir}/wikidump.conf.monitor" --rsynclists --relpath --outputdir
"${datadir}/public/" --toplevel
+python /usr/local/bin/list-last-n-good-dumps.py --dumpsnumber 1,2,3,4,5
--dirlisting 'rsync-dirlist-last-%s-good.txt' --configfile
"${confsdir}/wikidump.conf:monitor" --rsynclists --relpath --outputdir
"${datadir}/public/"
+python /usr/local/bin/list-last-n-good-dumps.py --dumpsnumber 1,2,3,4,5
--filelisting 'rsync-filelist-last-%s-good.txt' --configfile
"${confsdir}/wikidump.conf:monitor" --rsynclists --relpath --outputdir
"${datadir}/public/" --toplevel
# these lists can be used for rsync excl/incl on our side, providing shares
that "just work" for the mirrors
-python /usr/local/bin/list-last-n-good-dumps.py --dumpsnumber 1,2,3,4,5
--rsynclisting 'rsync-inc-last-%s.txt' --configfile
"${confsdir}/wikidump.conf.monitor" --relpath --outputdir "${datadir}/public/"
--toplevel
+python /usr/local/bin/list-last-n-good-dumps.py --dumpsnumber 1,2,3,4,5
--rsynclisting 'rsync-inc-last-%s.txt' --configfile
"${confsdir}/wikidump.conf:monitor" --relpath --outputdir "${datadir}/public/"
--toplevel
diff --git a/modules/snapshot/manifests/cron/pagetitles.pp
b/modules/snapshot/manifests/cron/pagetitles.pp
index e06376d..0df5588 100644
--- a/modules/snapshot/manifests/cron/pagetitles.pp
+++ b/modules/snapshot/manifests/cron/pagetitles.pp
@@ -20,7 +20,7 @@
ensure => 'present',
environment => '[email protected]',
user => $user,
- command => "cd ${repodir}; python onallwikis.py --configfile
${confsdir}/wikidump.conf.monitor --filenameformat
'{w}-{d}-all-titles-in-ns-0.gz' --outdir '${otherdir}/pagetitles/{d}' --query
\"'select page_title from page where page_namespace=0;'\"",
+ command => "cd ${repodir}; python onallwikis.py --configfile
${confsdir}/wikidump.conf:monitor --filenameformat
'{w}-{d}-all-titles-in-ns-0.gz' --outdir '${otherdir}/pagetitles/{d}' --query
\"'select page_title from page where page_namespace=0;'\"",
minute => '10',
hour => '8',
}
@@ -29,7 +29,7 @@
ensure => 'present',
environment => '[email protected]',
user => $user,
- command => "cd ${repodir}; python onallwikis.py --configfile
${confsdir}/wikidump.conf.monitor --filenameformat
'{w}-{d}-all-media-titles.gz' --outdir '${otherdir}/mediatitles/{d}' --query
\"'select page_title from page where page_namespace=6;'\"",
+ command => "cd ${repodir}; python onallwikis.py --configfile
${confsdir}/wikidump.conf:monitor --filenameformat
'{w}-{d}-all-media-titles.gz' --outdir '${otherdir}/mediatitles/{d}' --query
\"'select page_title from page where page_namespace=6;'\"",
minute => '50',
hour => '8',
}
diff --git a/modules/snapshot/manifests/dumps/configs.pp
b/modules/snapshot/manifests/dumps/configs.pp
index a6ffb47..24f66d1 100644
--- a/modules/snapshot/manifests/dumps/configs.pp
+++ b/modules/snapshot/manifests/dumps/configs.pp
@@ -12,170 +12,129 @@
$wikidatachunkhistory2 =
'2587436,951696,942913,837759,1568292,1293747,2018593,1461235,1797642,1487121,2012246,874850,1486799'
$config = {
- smallwikis => {
- dblist => "${apachedir}/dblists/all.dblist",
- skipdblist => "${dblistsdir}/skip.dblist",
- keep => '10',
- chunksEnabled => '0',
- },
- bigwikis => {
- dblist => "${dblistsdir}/bigwikis.dblist",
- skipdblist => "${dblistsdir}/skipnone.dblist",
- keep => '8',
- chunksEnabled => '1',
- recombineHistory => '0',
- revsPerJob => '1500000',
- retryWait => '30',
- maxRetries => '3',
- revsMargin => '100',
- wikis => {
- ruwiki => {
- pagesPerChunkHistory => '311181,1142420,1627923,3122803',
- pagesPerChunkAbstract => '1200000',
- chunksForAbstract => '4',
- checkpointTime => '720',
- },
- eswiki => {
- pagesPerChunkHistory => '229078,854374,2324057,3860337',
- pagesPerChunkAbstract => '1500000',
- chunksForAbstract => '4',
- checkpointTime => '720',
- },
- dewiki => {
- pagesPerChunkHistory => '425449,1451596,3087456,4274631',
- pagesPerChunkAbstract => '2000000',
- chunksForAbstract => '4',
- checkpointTime => '720',
- },
- ptwiki => {
- pagesPerChunkHistory => '158756,789502,1361027,2669897',
- pagesPerChunkAbstract => '1000000',
- chunksForAbstract => '4',
- },
- plwiki => {
- pagesPerChunkHistory => '235373,514490,957079,1923498',
- pagesPerChunkAbstract => '800000',
- chunksForAbstract => '4',
- },
- nlwiki => {
- pagesPerChunkHistory => '224145,552515,1098243,2570924',
- pagesPerChunkAbstract => '1000000',
- chunksForAbstract => '4',
- },
- frwiki => {
- pagesPerChunkHistory => '412303,1235589,2771967,5313378',
- pagesPerChunkAbstract => '1900000',
- chunksForAbstract => '4',
- checkpointTime => '720',
- },
- itwiki => {
- pagesPerChunkHistory => '442893,1049883,1381705,2929437',
- pagesPerChunkAbstract => '1200000',
- chunksForAbstract => '4',
- checkpointTime => '720',
- },
- jawiki => {
- pagesPerChunkHistory => '168815,480631,943865,1760565',
- pagesPerChunkAbstract => '800000',
- chunksForAbstract => '4',
- checkpointTime => '720',
- },
- commonswiki => {
- pagesPerChunkHistory =>
'6457504,9672260,12929298,17539944',
- pagesPerChunkAbstract => '11000000',
- chunksForAbstract => '4',
- checkpointTime => '720',
- },
- zhwiki => {
- pagesPerChunkHistory => '231819,564192,1300322,3112369',
- pagesPerChunkAbstract => '1300000',
- chunksForAbstract => '4',
- },
- metawiki => {
- pagesPerChunkHistory => '823386,2594392,3242670,3284506',
- pagesPerChunkAbstract => '2500000',
- chunksForAbstract => '4',
+ allwikis => {
+ global => {
+ dblist => "${apachedir}/dblists/all.dblist",
+ skipdblist => "${dblistsdir}/skip.dblist",
+ keep => '10',
+ chunksEnabled => '0',
+ },
+ big => {
+ dblist => "${dblistsdir}/bigwikis.dblist",
+ skipdblist => "${dblistsdir}/skipnone.dblist",
+ keep => '8',
+ chunksEnabled => '1',
+ chunksForAbstract => '4',
+ checkpointTime => '720',
+ recombineHistory => '0',
+ revsPerJob => '1500000',
+ retryWait => '30',
+ maxRetries => '3',
+ revsMargin => '100',
+ wikis => {
+ ruwiki => {
+ pagesPerChunkHistory =>
'311181,1142420,1627923,3122803',
+ pagesPerChunkAbstract => '1200000',
+ },
+ eswiki => {
+ pagesPerChunkHistory =>
'229078,854374,2324057,3860337',
+ pagesPerChunkAbstract => '1500000',
+ },
+ dewiki => {
+ pagesPerChunkHistory =>
'425449,1451596,3087456,4274631',
+ pagesPerChunkAbstract => '2000000',
+ },
+ ptwiki => {
+ pagesPerChunkHistory =>
'158756,789502,1361027,2669897',
+ pagesPerChunkAbstract => '1000000',
+ },
+ plwiki => {
+ pagesPerChunkHistory =>
'235373,514490,957079,1923498',
+ pagesPerChunkAbstract => '800000',
+ },
+ nlwiki => {
+ pagesPerChunkHistory =>
'224145,552515,1098243,2570924',
+ pagesPerChunkAbstract => '1000000',
+ },
+ frwiki => {
+ pagesPerChunkHistory =>
'412303,1235589,2771967,5313378',
+ pagesPerChunkAbstract => '1900000',
+ },
+ itwiki => {
+ pagesPerChunkHistory =>
'442893,1049883,1381705,2929437',
+ pagesPerChunkAbstract => '1200000',
+ },
+ jawiki => {
+ pagesPerChunkHistory =>
'168815,480631,943865,1760565',
+ pagesPerChunkAbstract => '800000',
+ },
+ commonswiki => {
+ pagesPerChunkHistory =>
'6457504,9672260,12929298,17539944',
+ pagesPerChunkAbstract => '11000000',
+ },
+ zhwiki => {
+ pagesPerChunkHistory =>
'231819,564192,1300322,3112369',
+ pagesPerChunkAbstract => '1300000',
+ },
+ metawiki => {
+ pagesPerChunkHistory =>
'823386,2594392,3242670,3284506',
+ pagesPerChunkAbstract => '2500000',
+ },
},
},
- },
- wikidatawiki => {
- dblist => "${dblistsdir}/wikidatawiki.dblist",
- skipdblist => "${dblistsdir}/skipnone.dblist",
- keep => '7',
- chunksEnabled => '1',
- recombineHistory => '0',
- checkpointTime => '720',
- revsPerJob => '1500000',
- retryWait => '30',
- maxRetries => '3',
- revsMargin => '100',
- wikis => {
- wikidatawiki => {
- jobsperbatch => 'xmlstubsdump=9,abstractsdump=9',
- pagesPerChunkHistory =>
"${wikidatachunkhistory1},${$wikidatachunkhistory2}",
- chunksForAbstract => 27,
- orderrevs => '1',
- minpages => '10',
- maxrevs => '20000',
+ en => {
+ dblist => "${dblistsdir}/enwiki.dblist",
+ skipdblist => "${dblistsdir}/skipnone.dblist",
+ jobsperbatch => 'xmlstubsdump=9,abstractsdump=9',
+ keep => '7',
+ chunksEnabled => '1',
+ chunksForAbstract => '27',
+ recombineHistory => '0',
+ checkpointTime => '720',
+ revsPerJob => '1500000',
+ retryWait => '30',
+ maxRetries => '3',
+ revsMargin => '100',
+ maxrevs => '20000',
+ wikis => {
+ enwiki => {
+ pagesPerChunkHistory =>
"${enchunkhistory1},${enchunkhistory2}",
+ },
},
},
- },
- enwiki => {
- dblist => "${dblistsdir}/enwiki.dblist",
- skipdblist => "${dblistsdir}/skipnone.dblist",
- keep => '7',
- chunksEnabled => '1',
- recombineHistory => '0',
- checkpointTime => '720',
- revsPerJob => '1500000',
- retryWait => '30',
- maxRetries => '3',
- revsMargin => '100',
- wikis => {
- enwiki => {
- jobsperbatch => 'xmlstubsdump=9,abstractsdump=9',
- pagesPerChunkHistory =>
"${enchunkhistory1},${enchunkhistory2}",
- chunksForAbstract => 27,
- maxrevs => '20000',
+ wd => {
+ dblist => "${dblistsdir}/wikidatawiki.dblist",
+ skipdblist => "${dblistsdir}/skipnone.dblist",
+ jobsperbatch => 'xmlstubsdump=9,abstractsdump=9',
+ keep => '7',
+ chunksEnabled => '1',
+ chunksForAbstract => '27',
+ recombineHistory => '0',
+ checkpointTime => '720',
+ revsPerJob => '1500000',
+ retryWait => '30',
+ maxRetries => '3',
+ revsMargin => '100',
+ maxrevs => '20000',
+ wikis => {
+ wikidatawiki => {
+ pagesPerChunkHistory =>
"${wikidatachunkhistory1},${$wikidatachunkhistory2}",
+ orderrevs => '1',
+ minpages => '10',
+ },
},
},
- },
- monitor => {
- dblist => "${apachedir}/dblists/all.dblist",
- skipdblist => "${dblistsdir}/skipmonitor.dblist",
- keep => '30',
- chunksEnabled => '0',
- },
- media => {
- dblist => "${apachedir}/dblists/all.dblist",
- skipdblist =>
"${dblistsdir}/skipmonitor.dblist,${dblistsdir}/globalusage.dblist",
- keep => '30',
- chunksEnabled => '0',
+ monitor => {
+ skipdblist => "${dblistsdir}/skipmonitor.dblist",
+ },
+ media => {
+ skipdblist =>
"${dblistsdir}/skipmonitor.dblist,${dblistsdir}/globalusage.dblist",
+ }
},
}
snapshot::dumps::wikiconf { 'wikidump.conf':
- configtype => 'smallwikis',
- config => $config,
- }
- snapshot::dumps::wikiconf { 'wikidump.conf.bigwikis':
- configtype => 'bigwikis',
- config => $config,
- }
- snapshot::dumps::wikiconf { 'wikidump.conf.enwiki':
- configtype => 'enwiki',
- config => $config,
- }
- snapshot::dumps::wikiconf { 'wikidump.conf.wikidatawiki':
- configtype => 'wikidatawiki',
- config => $config,
- }
- snapshot::dumps::wikiconf { 'wikidump.conf.monitor':
- configtype => 'monitor',
- config => $config,
- }
- snapshot::dumps::wikiconf { 'wikidump.conf.media':
- configtype => 'media',
+ configtype => 'allwikis',
config => $config,
}
diff --git a/modules/snapshot/manifests/dumps/monitor.pp
b/modules/snapshot/manifests/dumps/monitor.pp
index 7532136..59e79e1 100644
--- a/modules/snapshot/manifests/dumps/monitor.pp
+++ b/modules/snapshot/manifests/dumps/monitor.pp
@@ -8,6 +8,6 @@
ensure => 'present',
systemd => systemd_template('dumps-monitor'),
upstart => upstart_template('dumps-monitor'),
- subscribe => File["${confsdir}/wikidump.conf.monitor"],
+ subscribe => File["${confsdir}/wikidump.conf"],
}
}
diff --git a/modules/snapshot/manifests/dumps/stagesconfig.pp
b/modules/snapshot/manifests/dumps/stagesconfig.pp
index 9052faa..e13622d 100644
--- a/modules/snapshot/manifests/dumps/stagesconfig.pp
+++ b/modules/snapshot/manifests/dumps/stagesconfig.pp
@@ -8,9 +8,9 @@
$wikiargs = '/bin/bash ./worker --skipdone --exclusive --log'
$args_smallwikis = "${wikiargs} --configfile ${confsdir}/wikidump.conf"
- $args_bigwikis = "${wikiargs} --configfile
${confsdir}/wikidump.conf.bigwikis"
- $args_enwiki = "${wikiargs} --configfile ${confsdir}/wikidump.conf.enwiki"
- $args_wikidatawiki = "${wikiargs} --configfile
${confsdir}/wikidump.conf.wikidatawiki"
+ $args_bigwikis = "${wikiargs} --configfile
${confsdir}/wikidump.conf:bigwikis"
+ $args_enwiki = "${wikiargs} --configfile ${confsdir}/wikidump.conf:en"
+ $args_wikidatawiki = "${wikiargs} --configfile
${confsdir}/wikidump.conf:wd"
$jobs_to_skip = join(['metahistorybz2dump',
'metahistorybz2dumprecombine',
diff --git a/modules/snapshot/manifests/dumps/wikiconf.pp
b/modules/snapshot/manifests/dumps/wikiconf.pp
index 2e7af1d..ed8f894 100644
--- a/modules/snapshot/manifests/dumps/wikiconf.pp
+++ b/modules/snapshot/manifests/dumps/wikiconf.pp
@@ -1,5 +1,5 @@
define snapshot::dumps::wikiconf(
- $configtype = 'smallwikis',
+ $configtype = 'allwikis',
$config = undef,
) {
diff --git a/modules/snapshot/templates/dumps/fulldumps.sh.erb
b/modules/snapshot/templates/dumps/fulldumps.sh.erb
index c751ab9..f89be75 100644
--- a/modules/snapshot/templates/dumps/fulldumps.sh.erb
+++ b/modules/snapshot/templates/dumps/fulldumps.sh.erb
@@ -154,17 +154,17 @@
# (small, big, enwiki, wikidatawiki) as needed
case $wikitype in
'enwiki')
- maybe_do_createdirs "wikidump.conf.enwiki" "stages_create_enwiki"
"enwiki"
- maybe_do_createdirs "wikidump.conf.bigwikis" "stages_create_bigwikis"
"bigwikis"
+ maybe_do_createdirs "wikidump.conf:en" "stages_create_enwiki" "enwiki"
+ maybe_do_createdirs "wikidump.conf:bigwikis" "stages_create_bigwikis"
"bigwikis"
maybe_do_createdirs "wikidump.conf" "stages_create_smallwikis"
"smallwikis"
;;
'wikidatawiki')
- maybe_do_createdirs "wikidump.conf.wikidatawiki"
"stages_create_wikidatawiki" "wikidatawiki"
- maybe_do_createdirs "wikidump.conf.bigwikis" "stages_create_bigwikis"
"bigwikis"
+ maybe_do_createdirs "wikidump.conf:wd" "stages_create_wikidatawiki"
"wikidatawiki"
+ maybe_do_createdirs "wikidump.conf:bigwikis" "stages_create_bigwikis"
"bigwikis"
maybe_do_createdirs "wikidump.conf" "stages_create_smallwikis"
"smallwikis"
;;
'regular')
- maybe_do_createdirs "wikidump.conf.bigwikis" "stages_create_bigwikis"
"bigwikis"
+ maybe_do_createdirs "wikidump.conf:bigwikis" "stages_create_bigwikis"
"bigwikis"
maybe_do_createdirs "wikidump.conf" "stages_create_smallwikis"
"smallwikis"
;;
'*')
@@ -195,11 +195,11 @@
case $wikitype in
'enwiki')
- maybe_do_dumps "wikidump.conf.enwiki" "$enstages" "enwiki" "$maxjobs"
+ maybe_do_dumps "wikidump.conf:en" "$enstages" "enwiki" "$maxjobs"
# After enwiki is done, check and do some of the rest if needed
# If not all the big wikis are done we will start a dump run covering
small and big wikis
# Only wikis without complete dumps will be updated.
- maybe_do_dumps "wikidump.conf.bigwikis" "$regularstages" "bigwikis"
"$maxjobs"
+ maybe_do_dumps "wikidump.conf:bigwikis" "$regularstages" "bigwikis"
"$maxjobs"
if [ $? -ne 0 ]; then
# If we did not start a dump run above for small and big wikis,
and the small wikis
# are not done, start such a run now. Only wikis without complete
dumps will be updated.
@@ -207,11 +207,11 @@
fi
;;
'wikidatawiki')
- maybe_do_dumps "wikidump.conf.wikidatawiki" "$wikidatastages"
"wikidatawiki" "$maxjobs"
+ maybe_do_dumps "wikidump.conf:wd" "$wikidatastages" "wikidatawiki"
"$maxjobs"
# After wikidatawiki is done, check and do some of the rest if needed
# If not all the big wikis are done we will start a dump run covering
small and big wikis
# Only wikis without complete dumps will be updated.
- maybe_do_dumps "wikidump.conf.bigwikis" "$regularstages" "bigwikis"
"$maxjobs"
+ maybe_do_dumps "wikidump.conf:bigwikis" "$regularstages" "bigwikis"
"$maxjobs"
if [ $? -ne 0 ]; then
# If we did not start a dump run above for small and big wikis,
and the small wikis
# are not done, start such a run now. Only wikis without complete
dumps will be updated.
@@ -221,7 +221,7 @@
'regular')
# If not all the big wikis are done we will start a dump run covering
small and big wikis
# Only wikis without complete dumps will be updated.
- maybe_do_dumps "wikidump.conf.bigwikis" "$regularstages" "bigwikis"
"$maxjobs"
+ maybe_do_dumps "wikidump.conf:bigwikis" "$regularstages" "bigwikis"
"$maxjobs"
if [ $? -ne 0 ]; then
# If we did not start a dump run above for small and big wikis,
and the small wikis
# are not done, start such a run now. Only wikis without complete
dumps will be updated.
diff --git a/modules/snapshot/templates/dumps/wikidump.conf.erb
b/modules/snapshot/templates/dumps/wikidump.conf.erb
index 6e580d7..aece7f8 100644
--- a/modules/snapshot/templates/dumps/wikidump.conf.erb
+++ b/modules/snapshot/templates/dumps/wikidump.conf.erb
@@ -4,10 +4,10 @@
#############################################################
[wiki]
-dblist=<%= @config[@configtype]['dblist'] %>
+dblist=<%= @config[@configtype]['global']['dblist'] %>
privatelist=<%= scope.lookupvar('snapshot::dumps::dirs::apachedir')
-%>/dblists/private.dblist
closedlist=<%= scope.lookupvar('snapshot::dumps::dirs::apachedir')
-%>/dblists/closed.dblist
-skipdblist=<%= @config[@configtype]['skipdblist'] %>
+skipdblist=<%= @config[@configtype]['global']['skipdblist'] %>
flowlist=<%= scope.lookupvar('snapshot::dumps::dirs::apachedir')
-%>/dblists/flow.dblist
dir=<%= scope.lookupvar('snapshot::dumps::dirs::apachedir') %>
adminsettings=private/PrivateSettings.php
@@ -46,28 +46,28 @@
recompressxml=/usr/local/bin/recompressxml
[cleanup]
-keep=<%= @config[@configtype]['keep'] %>
+keep=<%= @config[@configtype]['global']['keep'] %>
[chunks]
-<% if @config[@configtype].has_key?('chunksEnabled') -%>
-chunksEnabled=<%= @config[@configtype]['chunksEnabled'] %>
+<% if @config[@configtype]['global'].has_key?('chunksEnabled') -%>
+chunksEnabled=<%= @config[@configtype]['global']['chunksEnabled'] %>
<% end -%>
-<% if @config[@configtype].has_key?('recombineHistory') -%>
+<% if @config[@configtype]['global'].has_key?('recombineHistory') -%>
recombineHistory=<%= @config[@configtype]['recombineHistory'] %>
<% end -%>
-<% if @config[@configtype].has_key?('checkpointTime') -%>
+<% if @config[@configtype]['global'].has_key?('checkpointTime') -%>
checkpointTime=<%= @config[@configtype]['checkpointTime'] %>
<% end -%>
-<% if @config[@configtype].has_key?('revsPerJob') -%>
+<% if @config[@configtype]['global'].has_key?('revsPerJob') -%>
revsPerJob=<%= @config[@configtype]['revsPerJob'] %>
<% end -%>
-<% if @config[@configtype].has_key?('retryWait') -%>
+<% if @config[@configtype]['global'].has_key?('retryWait') -%>
retryWait=<%= @config[@configtype]['retryWait'] %>
<% end -%>
-<% if @config[@configtype].has_key?('maxRetries') -%>
+<% if @config[@configtype]['global'].has_key?('maxRetries') -%>
maxRetries=<%= @config[@configtype]['maxRetries'] %>
<% end -%>
-<% if @config[@configtype].has_key?('revsMargin') -%>
+<% if @config[@configtype]['global'].has_key?('revsMargin') -%>
revsMargin=<%= @config[@configtype]['revsMargin'] %>
<% end -%>
@@ -79,27 +79,86 @@
minpages=1
maxrevs=100000
-<% if @config[@configtype].has_key?('wikis') %>
-<% @config[@configtype]['wikis'].keys.sort.each do |w| -%>
-[<%= w %>]
-<% if @config[@configtype]['wikis'][w].has_key?('jobsperbatch') -%>
-jobsperbatch=<%= @config[@configtype]['wikis'][w]['jobsperbatch'] %>
+[bigwikis]
+# generic settings for big wikis
+<% @config[@configtype]['big'].keys.sort.each do |bigsetting| -%>
+<% if bigsetting != 'wikis' -%>
+<%= bigsetting %>=<%= @config[@configtype]['big'][bigsetting] %>
<% end -%>
-pagesPerChunkHistory=<%=
@config[@configtype]['wikis'][w]['pagesPerChunkHistory'] %>
-pagesPerChunkAbstract=<%=
@config[@configtype]['wikis'][w]['pagesPerChunkAbstract'] %>
-chunksForAbstract=<%= @config[@configtype]['wikis'][w]['chunksForAbstract'] %>
-<% if @config[@configtype]['wikis'][w].has_key?('checkpointTime') -%>
-checkpointTime=<%= @config[@configtype]['wikis'][w]['checkpointTime'] %>
-<% end -%>
-<% if @config[@configtype]['wikis'][w].has_key?('orderrevs') -%>
-orderrevs=<%= @config[@configtype]['wikis'][w]['orderrevs'] %>
-<% end -%>
-<% if @config[@configtype]['wikis'][w].has_key?('minpages') -%>
-minpages=<%= @config[@configtype]['wikis'][w]['minpages'] %>
-<% end -%>
-<% if @config[@configtype]['wikis'][w].has_key?('maxrevs') -%>
-maxrevs=<%= @config[@configtype]['wikis'][w]['maxrevs'] %>
<% end -%>
+[en]
+# generic settings for enwiki (huge)
+<% @config[@configtype]['en'].keys.sort.each do |ensetting| -%>
+<% if ensetting != 'wikis' -%>
+<%= ensetting %>=<%= @config[@configtype]['en'][ensetting] %>
+<% end -%>
+<% end -%>
+
+[wd]
+# generic settings for wikidata (huge)
+<% @config[@configtype]['wd'].keys.sort.each do |wdsetting| -%>
+<% if wdsetting != 'wikis' -%>
+<%= wdsetting %>=<%= @config[@configtype]['wd'][wdsetting] %>
+<% end -%>
+<% end -%>
+
+[monitor]
+# generic settings for monitor service
+<% @config[@configtype]['monitor'].keys.sort.each do |monitorsetting| -%>
+<% if monitorsetting != 'wikis' -%>
+<%= monitorsetting %>=<%= @config[@configtype]['monitor'][monitorsetting] %>
+<% end -%>
+<% end -%>
+
+[media]
+# generic settings for media title cron jobs
+<% @config[@configtype]['media'].keys.sort.each do |mediasetting| -%>
+<% if mediasetting != 'wikis' -%>
+<%= mediasetting %>=<%= @config[@configtype]['media'][mediasetting] %>
+<% end -%>
+<% end -%>
+
+########################
+# wiki-specific settings
+
+<% if @config[@configtype].has_key?('big') -%>
+<% if @config[@configtype]['big'].has_key?('wikis') -%>
+<% @config[@configtype]['big']['wikis'].keys.sort.each do |w| -%>
+
+[<%= w %>]
+# specific settings for wiki <%= w %>
+<% @config[@configtype]['big']['wikis'][w].keys.sort.each do |bigwikisetting|
-%>
+<%= bigwikisetting %>=<%=
@config[@configtype]['big']['wikis'][w][bigwikisetting] %>
+<% end -%>
+<% end -%>
+<% end -%>
+<% end -%>
+
+
+<% if @config[@configtype].has_key?('en') -%>
+<% if @config[@configtype]['en'].has_key?('wikis') -%>
+<% @config[@configtype]['en']['wikis'].keys.sort.each do |w| -%>
+
+[<%= w %>]
+# specific settings for wiki <%= w %>
+<% @config[@configtype]['en']['wikis'][w].keys.sort.each do |ensetting| -%>
+<%= ensetting %>=<%= @config[@configtype]['en']['wikis'][w][ensetting] %>
+<% end -%>
+<% end -%>
+<% end -%>
+<% end -%>
+
+
+<% if @config[@configtype].has_key?('wd') -%>
+<% if @config[@configtype]['wd'].has_key?('wikis') -%>
+<% @config[@configtype]['wd']['wikis'].keys.sort.each do |w| -%>
+
+[<%= w %>]
+# specific settings for wiki <%= w %>
+<% @config[@configtype]['wd']['wikis'][w].keys.sort.each do |wdsetting| -%>
+<%= wdsetting %>=<%= @config[@configtype]['wd']['wikis'][w][wdsetting] %>
+<% end -%>
+<% end -%>
<% end -%>
<% end -%>
diff --git a/modules/snapshot/templates/initscripts/dumps-monitor.systemd.erb
b/modules/snapshot/templates/initscripts/dumps-monitor.systemd.erb
index ecd607e..3e6afe4 100644
--- a/modules/snapshot/templates/initscripts/dumps-monitor.systemd.erb
+++ b/modules/snapshot/templates/initscripts/dumps-monitor.systemd.erb
@@ -9,7 +9,7 @@
[Service]
User=datasets
WorkingDirectory=<%= @repodir %>
-ExecStart=/bin/bash <%= @repodir -%>/monitor --configfile <%= @confsdir
-%>/wikidump.conf.monitor --basedir <%= @repodir %>
+ExecStart=/bin/bash <%= @repodir -%>/monitor --configfile <%= @confsdir
-%>/wikidump.conf:monitor --basedir <%= @repodir %>
[Install]
WantedBy=multi-user.target
diff --git a/modules/snapshot/templates/initscripts/dumps-monitor.upstart.erb
b/modules/snapshot/templates/initscripts/dumps-monitor.upstart.erb
index 38f2073..e3e4ab2 100644
--- a/modules/snapshot/templates/initscripts/dumps-monitor.upstart.erb
+++ b/modules/snapshot/templates/initscripts/dumps-monitor.upstart.erb
@@ -14,4 +14,4 @@
chdir <%= @repodir %>
-exec /bin/bash <%= @repodir -%>/monitor --configfile <%= @confsdir
-%>/wikidump.conf.monitor --basedir <%= @repodir %>
+exec /bin/bash <%= @repodir -%>/monitor --configfile <%= @confsdir
-%>/wikidump.conf:monitor --basedir <%= @repodir %>
--
To view, visit https://gerrit.wikimedia.org/r/386388
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I6fc98e3152693b073591da1c582aa9cb9c9b1411
Gerrit-PatchSet: 11
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: ArielGlenn <[email protected]>
Gerrit-Reviewer: ArielGlenn <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
