jenkins-bot has submitted this change. (
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/816832 )
Change subject: [IMPR] decrease memory usage and improve processing speed
......................................................................
[IMPR] decrease memory usage and improve processing speed
- Use a generator instead of a list of pages to process. This decreases
memory usage a lot and also speeds up start time by giving up sorting
all pages.
- preload the page contents
- catch KeyboardInterrupt and leave the mean loop
- add look & feel of CurrentPageBot
- print execution time finally
Change-Id: Ib3572e1cf76ce898bc238b6d40688c286603cdd9
---
M scripts/archivebot.py
1 file changed, 19 insertions(+), 17 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/archivebot.py b/scripts/archivebot.py
index e92448c..19ab8cf 100755
--- a/scripts/archivebot.py
+++ b/scripts/archivebot.py
@@ -815,26 +815,22 @@
return
for template_name in templates:
- pagelist = []
tmpl = pywikibot.Page(site, template_name, ns=10)
- if not filename and not pagename:
- if namespace is not None:
- ns = [str(namespace)]
- else:
- ns = []
- pywikibot.output('Fetching template transclusions...')
- pagelist.extend(tmpl.getReferences(only_template_inclusion=True,
- follow_redirects=False,
- namespaces=ns))
if filename:
with open(filename) as f:
- for pg in f.readlines():
- pagelist.append(pywikibot.Page(site, pg, ns=10))
- if pagename:
- pagelist.append(pywikibot.Page(site, pagename, ns=3))
- pagelist.sort()
- for pg in pagelist:
- pywikibot.output('Processing {}'.format(pg))
+ gen = [pywikibot.Page(site, line, ns=10) for line in f]
+ elif pagename:
+ gen = [pywikibot.Page(site, pagename, ns=3)]
+ else:
+ ns = [str(namespace)] if namespace is not None else []
+ pywikibot.output('Fetching template transclusions...')
+ gen = tmpl.getReferences(only_template_inclusion=True,
+ follow_redirects=False,
+ namespaces=ns,
+ content=True)
+ for pg in gen:
+ pywikibot.info('\n\n>>> <<lightpurple>>{}<<default>> <<<'
+ .format(pg.title()))
# Catching exceptions, so that errors in one page do not bail out
# the entire process
try:
@@ -847,7 +843,13 @@
except Exception:
pywikibot.exception('Error occurred while processing page {}'
.format(pg))
+ except KeyboardInterrupt:
+ pywikibot.info('\nUser quit bot run...')
+ return
if __name__ == '__main__':
+ start = datetime.datetime.now()
main()
+ pywikibot.info('\nExecution time: {} seconds'
+ .format((datetime.datetime.now() - start).seconds))
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/816832
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Ib3572e1cf76ce898bc238b6d40688c286603cdd9
Gerrit-Change-Number: 816832
Gerrit-PatchSet: 5
Gerrit-Owner: Xqt <[email protected]>
Gerrit-Reviewer: D3r1ck01 <[email protected]>
Gerrit-Reviewer: Matěj Suchánek <[email protected]>
Gerrit-Reviewer: PotsdamLamb
Gerrit-Reviewer: Xqt <[email protected]>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
_______________________________________________
Pywikibot-commits mailing list -- [email protected]
To unsubscribe send an email to [email protected]
