Hi Massimo,
Thanks for looking into this. I just ran a few tests and I can confirm that
the leak does repro on Mac OS X El Capitan (10.11.6 (15G1004)) with the
latest version of pyDAL (16.9) and the HEAD and latest tags for web2py repo
(https://github.com/web2py/web2py/releases). The leak did not repro from
the last pypi version of web2py (2.1.1). I ran the program I listed in
PyCharm on Mac OSX and watched the memory usage using htop installed on the
Mac using Homebrew.
Here are the detailed results:
1. pyDAL 16.9
from pydal import DAL
def open_close_dal():
sql = DAL()
sql.close()
if __name__ == '__main__':
while True:
open_close_dal()
Memory usage at ~4 mins: 193M
(https://www.dropbox.com/s/vj5v6hqya4mot2v/Screenshot%202016-11-08%2011.43.28.png?dl=0)
Memory usage at ~11 mins: 432M
(https://www.dropbox.com/s/hac3monmy134b8f/Screenshot%202016-11-08%2011.50.51.png?dl=0)
2. web2py 2.1.1 (pip install web2py==2.1.1)
from gluon import DAL
def open_close_dal():
sql = DAL()
sql.close()
if __name__ == '__main__':
while True:
open_close_dal()
Memory usage at ~5 mins: 16.7M
(https://www.dropbox.com/s/yjy8vv7fy7i79b0/Screenshot%202016-11-08%2012.14.52.png?dl=0)
Memory usage at ~11 mins: 16.7M
(https://www.dropbox.com/s/49laoi3tk74qhvp/Screenshot%202016-11-08%2012.21.32.png?dl=0)
3. web2py cloned @latest tag from https://github.com/web2py/web2py.git and
copied the gluon directory to my project in PyCharm.
from gluon import DAL
def open_close_dal():
sql = DAL()
sql.close()
if __name__ == '__main__':
while True:
open_close_dal()
Memory usage at ~4 mins: 187M
(https://www.dropbox.com/s/5hx01xve1mpemg5/Screenshot%202016-11-08%2012.32.54.png?dl=0)
Memory usage at ~11 mins: 398M
(https://www.dropbox.com/s/wcnpz15cor4g698/Screenshot%202016-11-08%2012.40.32.png?dl=0)
Thanks,
Sukhmeet
On Monday, November 7, 2016 at 5:52:17 PM UTC-8, Massimo Di Pierro wrote:
>
> I run your code on a Mac and I was unable to reproduce the leak. I invite
> others to try with different OSes and DAL version. It will help debug it.
>
> On Monday, 7 November 2016 19:48:35 UTC-6, Massimo Di Pierro wrote:
>>
>> Thank you for your help. We will investigate....
>> Any chance you can try if the leak exists with the DAL in web2py stable?
>>
>> On Friday, 4 November 2016 21:34:52 UTC-5, Sukhmeet Toor wrote:
>>>
>>> Hi web2py-users,
>>>
>>> I have an application that runs on CherryPy (v8.1.2) on Ubuntu (14.04.5
>>> LTS) and python 2.7.6. I use standalone pyDAL (v16.09) to connect to my
>>> MySQL DB on AWS RDS. I'm seeing a constant memory leak on my servers which
>>> causes the process to use up 100% of RAM in a few hours and eventual die a
>>> painful death (no SWAP enabled; enabling SWAP pushes back certain death a
>>> little).
>>>
>>> I have managed to reproduce the leak with a very small loop that
>>> repeatedly opens and closes the DAL object. I confirmed that memory
>>> continues to grow using htop and will eventually end up using 100% of RAM.
>>> I thought it was an MySQL adapter specific problem but I was able to repro
>>> it with the default sqlite adapter as well.
>>>
>>> from pydal import DAL
>>>
>>> def open_close_dal():
>>> sql = DAL()
>>> sql.close()
>>>
>>> if __name__ == '__main__':
>>> while True:
>>> open_close_dal()
>>>
>>>
>>> I read some really old threads on this issue that suggested calling
>>> sql._adapter.close() or sql._adapter.close_all_instances() or del sql but
>>> it doesn't seem to make a difference. I've tried tracking down the memory
>>> leak and managed to confirm that it is the guts of the DAL that are
>>> leaking. One iteration of the loop, with pympler.tracker enabled reports
>>> these objects created:
>>>
>>>
>>> types | # objects | total size
>>> ============================================ | =========== | ============
>>> dict | 672 | 423.38 KB
>>> list | 3525 | 354.35 KB
>>> str | 2851 | 162.49 KB
>>> <class 'collections.OrderedDict | 129 | 135.02 KB
>>> code | 585 | 73.12 KB
>>> <class 'pydal.dialects.MetaDialect | 32 | 28.25 KB
>>> <class 'pydal.parsers.MetaParser | 17 | 15.01 KB
>>> type | 16 | 14.12 KB
>>> <class 'pydal.representers.MetaRepresenter | 16 | 14.12 KB
>>> tuple | 168 | 12.85 KB
>>> <class 'pydal.dialects.sqltype_for | 178 | 11.12 KB
>>> weakref | 89 | 7.65 KB
>>> int | 258 | 6.05 KB
>>> <class 'pydal.representers.for_type | 49 | 3.06 KB
>>> function (wrap) | 25 | 2.93 KB
>>>
>>>
>>> Is this a known issue? I've been pulling out my hair with this for a few
>>> days now and would love any insight into it.
>>>
>>>
>>> Thanks in advance,
>>>
>>> Sukhmeet
>>>
>>>
>>> p.s. Older threads on this topic, which no solution that worked for me:
>>>
>>> https://groups.google.com/forum/#!newtopic/web2py/web2py/hmsupVHdDHo
>>>
>>> http://stackoverflow.com/questions/5539722/standalone-web2py-dal-with-pyramid-requests-threads-and-memory
>>>
>>>
>>>
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
