[MediaWiki-commits] [Gerrit] labs...heritage[wikidata]: Allow skipping some configs during full harvest.
jenkins-bot has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/374403 )
Change subject: Allow skipping some configs during full harvest.
..
Allow skipping some configs during full harvest.
Adds the `-skip_wd` command line argument which skips all wikidata/sparql
harvesting.
Adds the `skip` config setting which, when set to true, skips that
particular config file from harvesting.
In both cases harvesting is allowed if a particular language-country
combination is provided.
Bug: T174334
Change-Id: I79cba7c655e68c6814838f59ba5b96a858a79505
---
M erfgoedbot/update_database.py
1 file changed, 6 insertions(+), 0 deletions(-)
Approvals:
Jean-Frédéric: Looks good to me, approved
jenkins-bot: Verified
diff --git a/erfgoedbot/update_database.py b/erfgoedbot/update_database.py
index c459085..be9b80f 100755
--- a/erfgoedbot/update_database.py
+++ b/erfgoedbot/update_database.py
@@ -503,6 +503,7 @@
countrycode = u''
lang = u''
fullUpdate = True
+skip_wd = False
daysBack = 2 # Default 2 days. Runs every night so can miss one night.
conn = None
cursor = None
@@ -518,6 +519,8 @@
daysBack = int(value)
elif option == u'-fullupdate': # does nothing since already default
fullUpdate = True
+elif option == u'-skip_wd':
+skip_wd = True
else:
raise Exception(
u'Bad parameters. Expected "-countrycode", "-langcode", '
@@ -547,6 +550,9 @@
u'be used together.')
else:
for (countrycode, lang), countryconfig in
mconfig.countries.iteritems():
+if (countryconfig.get('skip') or
+(skip_wd and (countryconfig.get('type') == 'sparql'))):
+continue
pywikibot.log(
u'Working on countrycode "%s" in language "%s"' % (
countrycode, lang))
--
To view, visit https://gerrit.wikimedia.org/r/374403
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I79cba7c655e68c6814838f59ba5b96a858a79505
Gerrit-PatchSet: 2
Gerrit-Project: labs/tools/heritage
Gerrit-Branch: wikidata
Gerrit-Owner: Lokal Profil
Gerrit-Reviewer: Jean-Frédéric
Gerrit-Reviewer: Lokal Profil
Gerrit-Reviewer: Multichill
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] labs...heritage[wikidata]: Allow skipping some configs during full harvest.
Lokal Profil has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/374403 )
Change subject: Allow skipping some configs during full harvest.
..
Allow skipping some configs during full harvest.
Adds the `-skip_wd` command line argument which skips all wikidata/sparql
harvesting.
Adds the `skip` config setting which, when set to true, skips that
particular config file from harvesting.
In both cases harvesting is allowed if a particular language-country
combination is provided.
Bug: T174334
Change-Id: I79cba7c655e68c6814838f59ba5b96a858a79505
---
M erfgoedbot/update_database.py
1 file changed, 6 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/labs/tools/heritage
refs/changes/03/374403/1
diff --git a/erfgoedbot/update_database.py b/erfgoedbot/update_database.py
index c459085..2a112fb 100755
--- a/erfgoedbot/update_database.py
+++ b/erfgoedbot/update_database.py
@@ -503,6 +503,7 @@
countrycode = u''
lang = u''
fullUpdate = True
+skip_wd_harvest = False
daysBack = 2 # Default 2 days. Runs every night so can miss one night.
conn = None
cursor = None
@@ -518,6 +519,8 @@
daysBack = int(value)
elif option == u'-fullupdate': # does nothing since already default
fullUpdate = True
+elif option == u'-skip_wd':
+skip_wd_harvest = True
else:
raise Exception(
u'Bad parameters. Expected "-countrycode", "-langcode", '
@@ -547,6 +550,9 @@
u'be used together.')
else:
for (countrycode, lang), countryconfig in
mconfig.countries.iteritems():
+if ((skip_wd_harvest and (countryconfig.get('type') == 'sparql'))
+or countryconfig.get('skip')):
+continue
pywikibot.log(
u'Working on countrycode "%s" in language "%s"' % (
countrycode, lang))
--
To view, visit https://gerrit.wikimedia.org/r/374403
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I79cba7c655e68c6814838f59ba5b96a858a79505
Gerrit-PatchSet: 1
Gerrit-Project: labs/tools/heritage
Gerrit-Branch: wikidata
Gerrit-Owner: Lokal Profil
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
