jenkins-bot has submitted this change. (
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/789587 )
Change subject: [IMPR] Increase minimum workers
......................................................................
[IMPR] Increase minimum workers
doubling the workers to have one free for each family is a good value
Change-Id: I5fa697c8959ed5c70e8a76cf684981ff53dac55f
---
M scripts/maintenance/preload_sites.py
1 file changed, 7 insertions(+), 4 deletions(-)
Approvals:
Mpaa: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/maintenance/preload_sites.py
b/scripts/maintenance/preload_sites.py
index 3c0afcd..5e0743b 100755
--- a/scripts/maintenance/preload_sites.py
+++ b/scripts/maintenance/preload_sites.py
@@ -73,15 +73,18 @@
.. versionchanged:: 7.3
Default of worker is calculated like for Python 3.8 but preserves
- at least one worker more than families_list elements to ensure a
- worker can be added in :func:`preload_family`.
+ at least one worker for each element in families_list for better
+ performance.
"""
start = datetime.now()
if worker is None:
# Python 3.8 default
worker = min(32, (os.cpu_count() or 1) + 4)
- # allow to add futures in preload_family
- worker = max(len(families) + 1, worker)
+ # to allow adding futures in preload_family the workers must be one
+ # more than families are handled
+ worker = max(len(families) * 2, worker)
+ pywikibot.output('Using {} workers to process {} families'
+ .format(worker, len(families)))
with ThreadPoolExecutor(worker) as executor:
futures = {executor.submit(preload_family, family, executor)
for family in families}
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/789587
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I5fa697c8959ed5c70e8a76cf684981ff53dac55f
Gerrit-Change-Number: 789587
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <[email protected]>
Gerrit-Reviewer: D3r1ck01 <[email protected]>
Gerrit-Reviewer: Mpaa <[email protected]>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
_______________________________________________
Pywikibot-commits mailing list -- [email protected]
To unsubscribe send an email to [email protected]
