[pypy-dev] Continual Increase In Memory Utilization Using PyPy 6.0 (python 2.7) -- Help!?

2019-03-28 Thread Robert Whitcher
So I have a process that use PyPy and pymongo in a loop. It does basically the same thing every loop, which query a table in via pymongo and do a few non-save calculations and then wait and loop again The RSS of the process continually increased (the PYPY_GC_MAX is set pretty high). So I hooked in

Re: [pypy-dev] Continual Increase In Memory Utilization Using PyPy 6.0 (python 2.7) -- Help!?

2019-03-28 Thread Antonio Cuni
Hi Robert, are you using any package which relies on cpyext? I.e., modules written in C and/or with Cython (cffi is fine). IIRC, at the moment PyPy doesn't detect GC cycles which involve cpyext objects. So if you have a cycle which does e.g. Py_foo -> C_bar -> Py_foo (where Py_foo is a pure-pyt

Re: [pypy-dev] Continual Increase In Memory Utilization Using PyPy 6.0 (python 2.7) -- Help!?

2019-03-28 Thread Robert Whitcher
Thanks Antonio... Relative to cpyext... I am not sure. We are not directly, but who knows what is being used by incorporated modules (pymongo, etc..) Trying to create a stripped down test case... I filed bug here with some updates: https://bitbucket.org/pypy/pypy/issues/2982/continual-increase-i

Re: [pypy-dev] Continual Increase In Memory Utilization Using PyPy 6.0 (python 2.7) -- Help!?

2019-03-28 Thread René Dudfield
Tried deleting the pymongo._cmessage module? It’s a cpyext one, so seems like a good place to start. I think it works without it. A general bisection memory leak finding technique that works in 11 runs or under... Run the code under coverage, so you have the lines that are actually called. Now com