Lokal Profil has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/374403 )
Change subject: Allow skipping some configs during full harvest.
......................................................................
Allow skipping some configs during full harvest.
Adds the `-skip_wd` command line argument which skips all wikidata/sparql
harvesting.
Adds the `skip` config setting which, when set to true, skips that
particular config file from harvesting.
In both cases harvesting is allowed if a particular language-country
combination is provided.
Bug: T174334
Change-Id: I79cba7c655e68c6814838f59ba5b96a858a79505
---
M erfgoedbot/update_database.py
1 file changed, 6 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/tools/heritage
refs/changes/03/374403/1
diff --git a/erfgoedbot/update_database.py b/erfgoedbot/update_database.py
index c459085..2a112fb 100755
--- a/erfgoedbot/update_database.py
+++ b/erfgoedbot/update_database.py
@@ -503,6 +503,7 @@
countrycode = u''
lang = u''
fullUpdate = True
+ skip_wd_harvest = False
daysBack = 2 # Default 2 days. Runs every night so can miss one night.
conn = None
cursor = None
@@ -518,6 +519,8 @@
daysBack = int(value)
elif option == u'-fullupdate': # does nothing since already default
fullUpdate = True
+ elif option == u'-skip_wd':
+ skip_wd_harvest = True
else:
raise Exception(
u'Bad parameters. Expected "-countrycode", "-langcode", '
@@ -547,6 +550,9 @@
u'be used together.')
else:
for (countrycode, lang), countryconfig in
mconfig.countries.iteritems():
+ if ((skip_wd_harvest and (countryconfig.get('type') == 'sparql'))
+ or countryconfig.get('skip')):
+ continue
pywikibot.log(
u'Working on countrycode "%s" in language "%s"' % (
countrycode, lang))
--
To view, visit https://gerrit.wikimedia.org/r/374403
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I79cba7c655e68c6814838f59ba5b96a858a79505
Gerrit-PatchSet: 1
Gerrit-Project: labs/tools/heritage
Gerrit-Branch: wikidata
Gerrit-Owner: Lokal Profil <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
