jenkins-bot has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/375448 )
Change subject: Make _flush aware of _putthread ongoing tasks
......................................................................
Make _flush aware of _putthread ongoing tasks
Make sure _flush() is aware if _putthread() is idle or working on a
task.
Relying on page_put_queue.qsize() > 0 only, as a condition to keep the while
loop alive is not safe as the last item in the queue can be fetched by
async_manager() before page_put_queue.qsize() is checked and then found empty.
In this case _flush() returns before all pending requests are served.
A new Queue is introduced as a form of synchronisation between
MainThread in _flush() and asyc_manager(), signalling when async_manager
is actually working and not idle waiting for a request.
Bug: T147178
Change-Id: I184b445c43aa44cb000bfacc1cad18d2de265c48
---
M pywikibot/__init__.py
1 file changed, 7 insertions(+), 1 deletion(-)
Approvals:
Zhuyifei1999: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py
index 9c9632d..6fe82cc 100644
--- a/pywikibot/__init__.py
+++ b/pywikibot/__init__.py
@@ -1372,7 +1372,9 @@
'{lightblue}Waiting for {num} pages to be put. '
'Estimated time remaining: {sec}{default}', num=num, sec=sec))
- while _putthread.is_alive() and page_put_queue.qsize() > 0:
+ while (_putthread.is_alive()
+ and (page_put_queue.qsize() > 0
+ or page_put_queue_busy.qsize() > 0)):
try:
_putthread.join(1)
except KeyboardInterrupt:
@@ -1398,10 +1400,12 @@
"""Daemon; take requests from the queue and execute them in background."""
while True:
(request, args, kwargs) = page_put_queue.get()
+ page_put_queue_busy.put(None)
if request is None:
break
request(*args, **kwargs)
page_put_queue.task_done()
+ page_put_queue_busy.get()
def async_request(request, *args, **kwargs):
@@ -1420,6 +1424,8 @@
# queue to hold pending requests
page_put_queue = Queue(config.max_queue_size)
+# queue to signal that async_manager is working on a request. See T147178.
+page_put_queue_busy = Queue(config.max_queue_size)
# set up the background thread
_putthread = threading.Thread(target=async_manager)
# identification for debugging purposes
--
To view, visit https://gerrit.wikimedia.org/r/375448
To unsubscribe, or for help writing mail filters, visit
https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I184b445c43aa44cb000bfacc1cad18d2de265c48
Gerrit-Change-Number: 375448
Gerrit-PatchSet: 4
Gerrit-Owner: Mpaa <[email protected]>
Gerrit-Reviewer: Dalba <[email protected]>
Gerrit-Reviewer: Dvorapa <[email protected]>
Gerrit-Reviewer: Framawiki <[email protected]>
Gerrit-Reviewer: John Vandenberg <[email protected]>
Gerrit-Reviewer: Magul <[email protected]>
Gerrit-Reviewer: Matěj Suchánek <[email protected]>
Gerrit-Reviewer: Merlijn van Deen <[email protected]>
Gerrit-Reviewer: Mpaa <[email protected]>
Gerrit-Reviewer: Xqt <[email protected]>
Gerrit-Reviewer: Zhuyifei1999 <[email protected]>
Gerrit-Reviewer: jenkins-bot (75)
_______________________________________________
Pywikibot-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/pywikibot-commits
