| Dvorapa renamed this task from "weblinkchecker.py slows down (itself, OS) to freeze after a while reaching 100% of CPU usage" to "weblinkchecker.py slows down (itself, OS) to freeze after a while reaching 100% of RAM". Dvorapa updated the task description. (Show Details) |
CHANGES TO TASK DESCRIPTION
Yes, I can set my OS not to reach 100% (still slows the whole OS after a while). Yes, I can limit weblinkchecker (pywikibot) to use only 50% CPU usagesome memory max (OS and CPU okRAM ok, but weblinkchecker.py still slows and freezes itself). Yes, I can interrupt it after a while and start it from the last article checked (currently the best solution I think, I use `timeout --signal=SIGINT 20m python pwb.py weblinkchecker -ns:0 -start:"Last article processed in last run"` and hope the data will not be broken at the end). But I think there is some issue in weblinkchecker.py, maybe duplicite calling of threads or functions; or threads or functions not properly terminated after a success or fail; or maybe functions or threads exceeds some limits which should not. And this issue makes weblinkchecker.py slowly grow until it reaches the cutoff and freezes.
...
weblinkchecker.py after some amount of time/pages read begins to slow itself down (let's say 1 hour). After another amount of time (2 hours) it slows also the whole OS by increasing CPURAM usage up to 100% (4 hours). It finally stops on 100% CPURAM usage when processing some page (`>>> Some page <<<` is the last what it outputs) and does nothing more (freezes). The whole OS (per CPURAM usage) is making itself slower until it freezes too. If I interrupt it by keyboard when CPURAM usage less than 100%, it usually says: `Waiting for remaining 49 threads to finish, please wait...`; on 100% CPURAM usage I can not do anything else than hard shutdown my PC (100% CPURAM usage makes the whole OS freeze).Yes, I can set my OS not to reach 100% (still slows the whole OS after a while). Yes, I can limit weblinkchecker (pywikibot) to use only 50% CPU usagesome memory max (OS and CPU okRAM ok, but weblinkchecker.py still slows and freezes itself). Yes, I can interrupt it after a while and start it from the last article checked (currently the best solution I think, I use `timeout --signal=SIGINT 20m python pwb.py weblinkchecker -ns:0 -start:"Last article processed in last run"` and hope the data will not be broken at the end). But I think there is some issue in weblinkchecker.py, maybe duplicite calling of threads or functions; or threads or functions not properly terminated after a success or fail; or maybe functions or threads exceeds some limits which should not. And this issue makes weblinkchecker.py slowly grow until it reaches the cutoff and freezes.
...
TASK DETAIL
EMAIL PREFERENCES
To: Dvorapa
Cc: Xqt, Zoranzoki21, zhuyifei1999, Aklapper, pywikibot-bugs-list, Dvorapa, Magul, Tbscho, rafidaslam, MayS, Mdupont, JJMC89, Avicennasis, jayvdb, Dalba, Masti, Alchimista, Rxy
Cc: Xqt, Zoranzoki21, zhuyifei1999, Aklapper, pywikibot-bugs-list, Dvorapa, Magul, Tbscho, rafidaslam, MayS, Mdupont, JJMC89, Avicennasis, jayvdb, Dalba, Masti, Alchimista, Rxy
_______________________________________________ pywikibot-bugs mailing list pywikibot-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
