| zhuyifei1999 added a comment. |
In T199615#4425451, @Xqt wrote:I see the getsizeof() counts the pointers only but not the Page objects itself.
The underlying implementation of set.__sizeof__ looks weird to me. It doesn't seem to be iterative or recursive on a first glance.
But yes, the problem is the container gets really large, and Page objects are too large to be kept from being garbage collected, and hashes are much better in this regard. Ideally we should have an upper bound to memory consumption, but IMO that's not really do-able in filter_unique without potential breakage.
TASK DETAIL
EMAIL PREFERENCES
To: zhuyifei1999
Cc: zhuyifei1999, Xqt, Aklapper, matej_suchanek, pywikibot-bugs-list, Magul, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
Cc: zhuyifei1999, Xqt, Aklapper, matej_suchanek, pywikibot-bugs-list, Magul, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy
_______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
