zhuyifei1999 created this task.
zhuyifei1999 added a project: Pywikibot.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper.
TASK DESCRIPTION
BasePage.botMayEdit is annotated with `@cache`
@cache
def botMayEdit(self) -> bool:
This is an unbounded strongref cache, so whenever `botMayEdit` is called,
`self` will be added to the cache as key, and the key will never be deleted
from cache. The `self` will then go on to reference way too many objects and
keeping them alive. One of my long-running bots used up 4 gigabytes of memory
before hitting rlimit and getting killed.
I tested that script with guppy3 to visualize just how much is being kept
alive by botMayEdit cache: (`dominos` is "The set 'dominated' by the set of
objects in x. This is the objects that will become deallocated, directly or
indirectly, when the objects in x are deallocated.")
When the bot first starts:
>>> import guppy
>>> hp = guppy.hpy()
>>> hp.heap()
Partition of a set of 270151 objects. Total size = 32191012 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 114602 42 9772353 30 9772353 30 str
1 10927 4 5288952 16 15061305 47 dict (no owner)
2 42074 16 4030904 13 19092209 59 tuple
3 20385 8 1538948 5 20631157 64 bytes
4 1565 1 1524032 5 22155189 69 type
5 10142 4 1464520 5 23619709 73 types.CodeType
6 9655 4 1390320 4 25010029 78 function
7 27941 10 796952 2 25806981 80 int
8 1565 1 790760 2 26597741 83 dict of type
9 5001 2 690752 2 27288493 85 list
<552 more rows. Type e.g. '_.more' to view.>
>>> hp.iso(pywikibot.page.BasePage.botMayEdit).dominos
Partition of a set of 11848 objects. Total size = 1652473 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 2474 21 752160 46 752160 46 dict (no owner)
1 6271 53 498823 30 1250983 76 str
2 1145 10 95872 6 1346855 82 list
3 150 1 64576 4 1411431 85 collections.OrderedDict
4 146 1 56016 3 1467447 89 dict of pywikibot.page.Page
5 193 2 42328 3 1509775 91 dict of pywikibot.page.Link
6 158 1 36656 2 1546431 94 set
7 85 1 21080 1 1567511 95 dict of pywikibot.page.Claim
8 265 2 19480 1 1586991 96 tuple
9 46 0 13040 1 1600031 97 dict of pywikibot.page.ItemPage
<28 more rows. Type e.g. '_.more' to view.>
>>> pywikibot.page.BasePage.botMayEdit.cache_info()
CacheInfo(hits=0, misses=5, maxsize=None, currsize=5)
After running for a few minutes:
>>> hp.heap()
Partition of a set of 1603419 objects. Total size = 208454157 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 230842 14 71763136 34 71763136 34 dict (no owner)
1 840630 52 60144617 29 131907753 63 str
2 129444 8 13086752 6 144994505 70 list
3 36258 2 13024648 6 158019153 76 collections.OrderedDict
4 19909 1 7455544 4 165474697 79 dict of pywikibot.page.Link
5 28394 2 7041712 3 172516409 83 dict of pywikibot.page.Claim
6 52982 3 4815056 2 177331465 85 tuple
7 12188 1 3108712 1 180440177 87 dict of pywikibot.page.ItemPage
8 7112 0 2737112 1 183177289 88 dict of pywikibot.page.Page
9 8177 1 2124904 1 185302193 89 set
<557 more rows. Type e.g. '_.more' to view.>
>>> hp.iso(pywikibot.page.BasePage.botMayEdit).dominos
Partition of a set of 1049673 objects. Total size = 141565142 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 193668 18 53690232 38 53690232 38 dict (no owner)
1 500122 48 33602128 24 87292360 62 str
2 35999 3 12842600 9 100134960 71 collections.OrderedDict
3 115539 11 10840032 8 110974992 78 list
4 19908 2 7455168 5 118430160 84 dict of pywikibot.page.Link
5 28394 3 7041712 5 125471872 89 dict of pywikibot.page.Claim
6 12188 1 3108712 2 128580584 91 dict of pywikibot.page.ItemPage
7 7111 1 2736736 2 131317320 93 dict of pywikibot.page.Page
8 28394 3 1817216 1 133134536 94 pywikibot.page.Claim
9 7567 1 1755544 1 134890080 95 set
<42 more rows. Type e.g. '_.more' to view.>
>>> pywikibot.page.BasePage.botMayEdit.cache_info()
CacheInfo(hits=0, misses=225, maxsize=None, currsize=225)
This is since rPWBC8b952048ab61: [bugfix] cache botMayEdit result
<https://phabricator.wikimedia.org/rPWBC8b952048ab61cbc2b78edf9ff647a5fcd1f1f535>
/ T267770: wikibase_tests fails for TestLoadRevisionsCaching.test_page_text
<https://phabricator.wikimedia.org/T267770>
TASK DETAIL
https://phabricator.wikimedia.org/T283957
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: zhuyifei1999
Cc: Aklapper, pywikibot-bugs-list, Xqt, zhuyifei1999, Shalomori123, Jyoo1011,
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS,
Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti,
Alchimista
_______________________________________________
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org
