zhuyifei1999 added a comment.
One way to solve this without using more lambdas:
diff --git a/pywikibot/comms/eventstreams.py b/pywikibot/comms/eventstreams.py
index a74baa1..b48ecc8 100644
--- a/pywikibot/comms/eventstreams.py
+++ b/pywikibot/comms/eventstreams.py
@@ -219,16 +219,16 @@ class
Dvorapa removed a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: DvorapaCc: gerritbot, Dalba, Xqt, Zoranzoki21, zhuyifei1999, Aklapper, pywikibot-bugs-list, Dvorapa, Magul,
Dvorapa added a comment.
In T185561#4020362, @zhuyifei1999 wrote:
Oh, if someone has the time to provide a core dump, along with gdb symbols, when the script memory usage is really high, I might be, unlikely, able to figure something out the root cause. Right now I really lack the time to babysit
Kippenvlees1 added a comment.
Problem seems to have been fixed. I haven't noticed this error pop up on the Dutch Wikipedia for a few months and found no Global User Pages lingering on the sample I just did on the top 5 wikis. I think this can be marked as resolved.TASK
IKhitron added a comment.
In T90978#4020822, @Kippenvlees1 wrote:
Problem seems to have been fixed. I haven't noticed this error pop up on the Dutch Wikipedia for a few months and found no Global User Pages lingering on the sample I just did on the top 5 wikis. I think this can be marked as
zhuyifei1999 added a comment.
In T185561#4020983, @Dvorapa wrote:
I have a partition to test install of Linux distros or similar stuff, I could install Ubuntu 17.10 with Python 3.6 and pywikibot in there and try gdb py-bt ;)
Thanks.TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL
Xqt added subscribers: Ottomata, Dalba, valhallasw.Xqt added a comment.
I know the solution (appending the function by another method like
def do_it:
self.filter[ftype].append(lambda e: key in e and e[key] == value)
for key, value in kwargs:
do_it(ftype, key, value)
) but I have no glue
Xqt closed this task as "Declined".Xqt added a comment.
rcstream is removedTASK DETAILhttps://phabricator.wikimedia.org/T102201EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jayvdb, XqtCc: Xqt, Tokenzero, Aklapper, jayvdb, pywikibot-bugs-list, Cpaulf30,
zhuyifei1999 added a comment.
In T188832#4021015, @Xqt wrote:
I have no glue why the current implementation doesn't work.
Variable scopes.
diff --git a/pywikibot/comms/eventstreams.py b/pywikibot/comms/eventstreams.py
index a74baa1..e5ccd9c 100644
--- a/pywikibot/comms/eventstreams.py
+++
gerritbot added a comment.
Change 416239 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] Enable EventStreams filter for multiple arguments
https://gerrit.wikimedia.org/r/416239TASK DETAILhttps://phabricator.wikimedia.org/T188832EMAIL
gerritbot added a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T188832EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Xqt, gerritbotCc: gerritbot, Mpaa, zhuyifei1999, valhallasw, Dalba, Ottomata, Aklapper, Zoranzoki21, Xqt,
Xaosflux added a subscriber: matej_suchanek.Xaosflux added a comment.
@matej_suchanek could this be the same issue you worked on for T161783TASK DETAILhttps://phabricator.wikimedia.org/T188846EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: XaosfluxCc:
zhuyifei1999 added a comment.
Made a breakpoint on where it sets blocking IO: P6789 T185561 breakpoint on set blocking networking IO. In line 145, it says timeout is None. No wonder why it blocks forever.TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL
zhuyifei1999 edited projects, added MediaWiki-API; removed ApiFeatureUsage.
TASK DETAILhttps://phabricator.wikimedia.org/T188846EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: zhuyifei1999Cc: matej_suchanek, Xaosflux, Aklapper, alanajjar, pywikibot-bugs-list,
zhuyifei1999 added a comment.
Reported: https://github.com/mementoweb/py-memento-client/issues/23
As for the memory leak I saw earlier umm, takes forever to produce anything useful. I'll see if I can keep it running and attach gdb onto it when the memory usage is really large.
@Dvorapa And
Xqt triaged this task as "High" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T188832EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: XqtCc: Aklapper, Zoranzoki21, Xqt, pywikibot-bugs-list, Magul, Tbscho, rafidaslam, MayS, Mdupont, JJMC89,
Xqt created this task.Xqt added projects: Pywikibot-core, Pywikibot-General.Herald added subscribers: pywikibot-bugs-list, Zoranzoki21, Aklapper.
TASK DESCRIPTIONSample:
>>> from pywikibot.comms.eventstreams import EventStreams
>>> stream = EventStreams(stream='recentchange')
>>>
zhuyifei1999 added a comment.
See also https://stackoverflow.com/q/7546285. It's indeed a hidden trap :(TASK DETAILhttps://phabricator.wikimedia.org/T188832EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Xqt, zhuyifei1999Cc: zhuyifei1999, valhallasw, Dalba,
Mpaa added a comment.
I think it is because in
for key, value in kwargs.items():
# append function for singletons
if isinstance(value, (bool, type(None))):
self.filter[ftype].append(lambda e: key in e and e[key] is value)
...
the last key, value pair is
Mpaa added a comment.
OK, I didn't see zhuyifei1999 answered as well while I was writing :-)TASK DETAILhttps://phabricator.wikimedia.org/T188832EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Xqt, MpaaCc: Mpaa, zhuyifei1999, valhallasw, Dalba, Ottomata,
zhuyifei1999 added a comment.
zhuyifei1999@zhuyifei1999-ThinkPad-X260:~/T185561test$ strace python3 -c 'import requests; requests.head("https://www.icann.org/en/tlds/agreements/cat/cat-agreement-23sep05.htm")'
execve("/home/zhuyifei1999/T185561test/.venv/bin/python3", ["python3", "-c", "import
zhuyifei1999 added a comment.
Apparently, the 0-results-returned is because of the gapfilterredir=redirects. Compare with filter and without filter.TASK DETAILhttps://phabricator.wikimedia.org/T188843EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
zhuyifei1999 added a comment.
Pywikibot does not yet have any (python) API that utilizes mediawiki's generator=allredirects API. Does it make more sense to add a new function for this (more elegant), make site.allpages() use it when filterredir=True (improve existing scripts), or do the former and
Xaosflux added a comment.
See also T161783TASK DETAILhttps://phabricator.wikimedia.org/T188846EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: XaosfluxCc: Xaosflux, Aklapper, alanajjar, pywikibot-bugs-list, Jar, Magul, Tbscho, rafidaslam, MayS, Mdupont,
Jar added a comment.
@Xaosflux My problem is a bit different, if the pages (titles) less than 50 it's work if more than 50 i get errorTASK DETAILhttps://phabricator.wikimedia.org/T188846EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: JarCc: matej_suchanek,
zhuyifei1999 removed a project: MediaWiki-API.zhuyifei1999 claimed this task.zhuyifei1999 added a comment.
generator=allredirects works so much better.TASK DETAILhttps://phabricator.wikimedia.org/T188843EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
Riley_Huntley created this task.Riley_Huntley added a project: Pywikibot-core.Herald added subscribers: pywikibot-bugs-list, Aklapper.
TASK DESCRIPTIONRunning category_redirect.py on RyanBot commons:commons using PAWS - Python 3 results in the following error:
Checking 55 category redirect pages
gerritbot added a comment.
Change 416333 had a related patch set uploaded (by Zhuyifei1999; owner: Zhuyifei1999):
[pywikibot/core@master] category_redirect.py: convert dict_keys to list when iterating
https://gerrit.wikimedia.org/r/416333TASK DETAILhttps://phabricator.wikimedia.org/T188850EMAIL
gerritbot added a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T188850EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: zhuyifei1999, gerritbotCc: gerritbot, Aklapper, Riley_Huntley, pywikibot-bugs-list, Giuliamocci, Adrian1985,
Dvorapa added a comment.
↑ (missing comment:) I reproduced the bug on Ubuntu 17.10. This was captured in freeze ([[.lu]] article currently processed). In addition I also tried (gdb) generate-core-file (found a suggestion somewhere in python forums) which gave me some type of binary file as an
Riley_Huntley created this task.Riley_Huntley added projects: Pywikibot-core, Pywikibot-category.py.Herald added subscribers: pywikibot-bugs-list, Aklapper.
TASK DESCRIPTIONPAWS:~$ pwb.py category_redirect.py -v
Python 3.6.3 (default, Oct 3 2017, 21:45:48)
[GCC 7.2.0]
LOADING SITE commons:commons
zhuyifei1999 added a comment.
I got this reproduced with an Arch Linux image, strace shows the thread stuck at read(31,, and lsof says fd 31 is www.icann.org:443, but I'm not sure about the exact url. Where are the debug symbols...TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL
zhuyifei1999 added a comment.
In line 334 of the paste, the url is revealed to be https://www.icann.org/en/tlds/agreements/cat/cat-agreement-23sep05.htmTASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
Dvorapa added a comment.
P6786 gdb weblinkchecker -repeatTASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: DvorapaCc: gerritbot, Dalba, Xqt, Zoranzoki21, zhuyifei1999, Aklapper, pywikibot-bugs-list, Dvorapa,
zhuyifei1999 added a comment.
API sandbox link of above.
Debug log from the script invoke to the first of such message: P6787 T188843 debug logTASK DETAILhttps://phabricator.wikimedia.org/T188843EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: zhuyifei1999Cc:
zhuyifei1999 added a comment.
^ uh yeah, the stuck process is doing read, not poll, I just realized.TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: zhuyifei1999Cc: gerritbot, Dalba, Xqt, Zoranzoki21,
zhuyifei1999 added a comment.
Reproduction code (simulating weblinkchecker.py#L200):
strace python3 -c 'import datetime, memento_client; mc = memento_client.MementoClient(); mc.timegate_uri = "http://web.archive.org/web/";
zhuyifei1999 added a comment.
First test run... unable to reproduce. then I realized in your traceback there is memento_client. Installed it, still no. Ran to 7. návštěvní expedice (ISS) without problems, after which I terminated it.TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL
zhuyifei1999 added a comment.
Headers: {'Content-Type': 'application/x-www-form-urlencoded'}
URI: '/w/api.php'
Body:
Jar created this task.Jar added projects: Pywikibot-core, ApiFeatureUsage.Herald added subscribers: pywikibot-bugs-list, alanajjar, Aklapper.
TASK DESCRIPTIONHello, for some reason a lot of time i get limit API warning, my bot has bot flag on projects that i work in.
WARNING: API warning
zhuyifei1999 added a comment.
I got a full bt of that process with only the stuck thread backtraced: P6788. The (hacky) commands are:
root@T185561:/# (gdb python3 -p 1675 -ex 'thread apply 4 py-bt' < /dev/null; gdb python3 -p 1675 -ex 'thread apply 4 bt' < /dev/null; gdb python3 -p 1675 -ex
zhuyifei1999 added a comment.
I tested this reproduction code:
PYWIKIBOT2_NO_USER_CONFIG=1 strace python3 -c 'import pwb; from pywikibot.comms.http import fetch; fetch("https://www.icann.org/en/tlds/agreements/cat/cat-agreement-23sep05.htm", headers={"User-Agent":"python-requests/2.18.4",
zhuyifei1999 added a comment.
ioctl(4, FIONBIO, [0]) basically sets the socket into blocking IO mode, comparing with the previous ioctl(3, FIONBIO, [1]).TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
zhuyifei1999 added a comment.
Thread 2 is waiting on SSL socket at line 825 of your paste. Why doesn't it timeout? Other threads are waiting on the semaphore.TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
zhuyifei1999 added a comment.
The symbol tables we have are probably very different, and unless you can copy the symbol tables over as well, I guess I'd better try to compile that version myself.
I run Ubuntu 16.04 and cannot easily switch to other linux distros / versions without messing around
zhuyifei1999 added a comment.
Compiled:
zhuyifei1999@zhuyifei1999-ThinkPad-X260:~/T185561test$ PATH=`pwd`/.venv/bin:$PATH
zhuyifei1999@zhuyifei1999-ThinkPad-X260:~/T185561test$ wget 'https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tar.xz'
--2018-03-03 16:39:32--
zhuyifei1999 added a comment.
(Now trying to run LXC...)TASK DETAILhttps://phabricator.wikimedia.org/T185561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: zhuyifei1999Cc: gerritbot, Dalba, Xqt, Zoranzoki21, zhuyifei1999, Aklapper, pywikibot-bugs-list,
Riley_Huntley updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION...[[ https://pastebin.com/Uu89yzXp | and repeat... ]] (python 3.6) and [[ https://pastebin.com/nEsnHvMe | on my other bot... ]] (python 2.7)
Running PAWS, cannot get pywikibot-core category_redirect.py to work
zhuyifei1999 edited projects, added MediaWiki-API; removed Pywikibot-category.py.
TASK DETAILhttps://phabricator.wikimedia.org/T188843EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: zhuyifei1999Cc: zhuyifei1999, Chicocvenancio, Aklapper, Riley_Huntley,
49 matches
Mail list logo
