[Pywikipedia-bugs] [Maniphest] [Commented On] T185561: weblinkchecker.py slows down (itself, OS) to freeze after a while reaching 100% of RAM

[zhuyifei1999]

zhuyifei1999 added a comment.

So this code seems to be able to extract the appproximate size of cookies, if navigated to the right frame (thread 2 & a few py-up):

python
def objptr_getattr(obj, name):
    try:
        return dict_getitem(obj.get_attr_dict(), name)
    except KeyError:
        raise AttributeError

def dict_getitem(obj, name):
    for key, val in obj.iteritems():
        if key.proxyval(set()) == name:
            return val
    else:
        raise KeyError

def pyobj_size(obj, visited):
    if obj.as_address() in visited:
        return 0

    if isinstance(obj, PyUnicodeObjectPtr):
        return obj.char_width()
    elif isinstance(obj, PyBytesObjectPtr):
        return int_from_int(obj.field('ob_size'))
    elif isinstance(obj, HeapTypeObjectPtr):
        return pyobj_size(obj.get_attr_dict(), visited)
    elif isinstance(obj, (PyTupleObjectPtr, PyListObjectPtr)):
        return sum(pyobj_size(obj[i], visited) for i in range(int_from_int(obj.field('ob_size'))))
    elif isinstance(obj, PyDictObjectPtr):
        return sum(pyobj_size(key, visited) + pyobj_size(val, visited) for key, val in obj.iteritems())
    elif isinstance(obj, (PyNoneStructPtr, PyLongObjectPtr)):
        return sys.getsizeof(obj.proxyval(visited))
    else:
        raise RuntimeError(type(obj))

end

python print(pyobj_size(objptr_getattr(objptr_getattr(Frame.get_selected_python_frame().get_pyop().get_var_by_name('self')[0], 'cookies'), '_cookies'), set()))

(gdb) py-bt
Traceback (most recent call first):
  File "/home/pavel/.local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/pavel/.local/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url="" **kwargs)
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 387, in _http_process
    **http_request.kwargs)
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 473, in _enqueue
    _http_process(session, request)
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 516, in fetch
    request = _enqueue(uri, method, params, body, headers, **kwargs)
  File "./scripts/weblinkchecker.py", line 602, in run
    use_fake_user_agent=self._use_fake_user_agent)
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
(gdb) python
  >def objptr_getattr(obj, name):
  >    try:
  >        return dict_getitem(obj.get_attr_dict(), name)
  >    except KeyError:
  >        raise AttributeError
  >
  >def dict_getitem(obj, name):
  >    for key, val in obj.iteritems():
  >        if key.proxyval(set()) == name:
  >            return val
  >    else:
  >        raise KeyError
  >
  >def pyobj_size(obj, visited):
  >    if obj.as_address() in visited:
  >        return 0
  >
  >    if isinstance(obj, PyUnicodeObjectPtr):
  >        return obj.char_width()
  >    elif isinstance(obj, PyBytesObjectPtr):
  >        return int_from_int(obj.field('ob_size'))
  >    elif isinstance(obj, HeapTypeObjectPtr):
  >        return pyobj_size(obj.get_attr_dict(), visited)
  >    elif isinstance(obj, (PyTupleObjectPtr, PyListObjectPtr)):
  >        return sum(pyobj_size(obj[i], visited) for i in range(int_from_int(obj.field('ob_size'))))
  >    elif isinstance(obj, PyDictObjectPtr):
  >        return sum(pyobj_size(key, visited) + pyobj_size(val, visited) for key, val in obj.iteritems())
  >    elif isinstance(obj, (PyNoneStructPtr, PyLongObjectPtr)):
  >        return sys.getsizeof(obj.proxyval(visited))
  >    else:
  >        raise RuntimeError(type(obj))
  >
  >end
(gdb) 
(gdb) python print(pyobj_size(objptr_getattr(objptr_getattr(Frame.get_selected_python_frame().get_pyop().get_var_by_name('self')[0], 'cookies'), '_cookies'), set()))
380

Since the session is new, I guess it's not surprising that the cookies jar has basically nothing.

TASK DETAIL

https://phabricator.wikimedia.org/T185561

EMAIL PREFERENCES

https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: zhuyifei1999
Cc: gerritbot, Dalba, Xqt, Zoranzoki21, zhuyifei1999, Aklapper, pywikibot-bugs-list, Dvorapa, Magul, Tbscho, rafidaslam, MayS, Mdupont, JJMC89, Avicennasis, jayvdb, Masti, Alchimista, Rxy

_______________________________________________
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs