I understand this was fixed here: https://github.com/web2py/pydal/releases/tag/v16.11
On Tuesday, 8 November 2016 15:07:37 UTC-6, Sukhmeet Toor wrote: > > Hi Massimo, > > Thanks for looking into this. I just ran a few tests and I can confirm > that the leak does repro on Mac OS X El Capitan (10.11.6 (15G1004)) with > the latest version of pyDAL (16.9) and the HEAD and latest tags for web2py > repo (https://github.com/web2py/web2py/releases). The leak did not repro > from the last pypi version of web2py (2.1.1). I ran the program I listed in > PyCharm on Mac OSX and watched the memory usage using htop installed on the > Mac using Homebrew. > > Here are the detailed results: > 1. pyDAL 16.9 > > from pydal import DAL > > def open_close_dal(): > sql = DAL() > sql.close() > > if __name__ == '__main__': > while True: > open_close_dal() > > > Memory usage at ~4 mins: 193M ( > https://www.dropbox.com/s/vj5v6hqya4mot2v/Screenshot%202016-11-08%2011.43.28.png?dl=0 > ) > Memory usage at ~11 mins: 432M ( > https://www.dropbox.com/s/hac3monmy134b8f/Screenshot%202016-11-08%2011.50.51.png?dl=0 > ) > > 2. web2py 2.1.1 (pip install web2py==2.1.1) > > from gluon import DAL > > def open_close_dal(): > sql = DAL() > sql.close() > > if __name__ == '__main__': > while True: > open_close_dal() > > > Memory usage at ~5 mins: 16.7M > (https://www.dropbox.com/s/yjy8vv7fy7i79b0/Screenshot%202016-11-08%2012.14.52.png?dl=0) > Memory usage at ~11 mins: 16.7M > (https://www.dropbox.com/s/49laoi3tk74qhvp/Screenshot%202016-11-08%2012.21.32.png?dl=0) > > 3. web2py cloned @latest tag from https://github.com/web2py/web2py.git and > copied the gluon directory to my project in PyCharm. > > from gluon import DAL > > def open_close_dal(): > sql = DAL() > sql.close() > > if __name__ == '__main__': > while True: > open_close_dal() > > > Memory usage at ~4 mins: 187M > (https://www.dropbox.com/s/5hx01xve1mpemg5/Screenshot%202016-11-08%2012.32.54.png?dl=0) > Memory usage at ~11 mins: 398M > (https://www.dropbox.com/s/wcnpz15cor4g698/Screenshot%202016-11-08%2012.40.32.png?dl=0) > > Thanks, > Sukhmeet > > > > On Monday, November 7, 2016 at 5:52:17 PM UTC-8, Massimo Di Pierro wrote: >> >> I run your code on a Mac and I was unable to reproduce the leak. I invite >> others to try with different OSes and DAL version. It will help debug it. >> >> On Monday, 7 November 2016 19:48:35 UTC-6, Massimo Di Pierro wrote: >>> >>> Thank you for your help. We will investigate.... >>> Any chance you can try if the leak exists with the DAL in web2py stable? >>> >>> On Friday, 4 November 2016 21:34:52 UTC-5, Sukhmeet Toor wrote: >>>> >>>> Hi web2py-users, >>>> >>>> I have an application that runs on CherryPy (v8.1.2) on Ubuntu (14.04.5 >>>> LTS) and python 2.7.6. I use standalone pyDAL (v16.09) to connect to my >>>> MySQL DB on AWS RDS. I'm seeing a constant memory leak on my servers which >>>> causes the process to use up 100% of RAM in a few hours and eventual die a >>>> painful death (no SWAP enabled; enabling SWAP pushes back certain death a >>>> little). >>>> >>>> I have managed to reproduce the leak with a very small loop that >>>> repeatedly opens and closes the DAL object. I confirmed that memory >>>> continues to grow using htop and will eventually end up using 100% of RAM. >>>> I thought it was an MySQL adapter specific problem but I was able to repro >>>> it with the default sqlite adapter as well. >>>> >>>> from pydal import DAL >>>> >>>> def open_close_dal(): >>>> sql = DAL() >>>> sql.close() >>>> >>>> if __name__ == '__main__': >>>> while True: >>>> open_close_dal() >>>> >>>> >>>> I read some really old threads on this issue that suggested calling >>>> sql._adapter.close() or sql._adapter.close_all_instances() or del sql but >>>> it doesn't seem to make a difference. I've tried tracking down the memory >>>> leak and managed to confirm that it is the guts of the DAL that are >>>> leaking. One iteration of the loop, with pympler.tracker enabled reports >>>> these objects created: >>>> >>>> >>>> types | # objects | total size >>>> ============================================ | =========== | ============ >>>> dict | 672 | 423.38 KB >>>> list | 3525 | 354.35 KB >>>> str | 2851 | 162.49 KB >>>> <class 'collections.OrderedDict | 129 | 135.02 KB >>>> code | 585 | 73.12 KB >>>> <class 'pydal.dialects.MetaDialect | 32 | 28.25 KB >>>> <class 'pydal.parsers.MetaParser | 17 | 15.01 KB >>>> type | 16 | 14.12 KB >>>> <class 'pydal.representers.MetaRepresenter | 16 | 14.12 KB >>>> tuple | 168 | 12.85 KB >>>> <class 'pydal.dialects.sqltype_for | 178 | 11.12 KB >>>> weakref | 89 | 7.65 KB >>>> int | 258 | 6.05 KB >>>> <class 'pydal.representers.for_type | 49 | 3.06 KB >>>> function (wrap) | 25 | 2.93 KB >>>> >>>> >>>> Is this a known issue? I've been pulling out my hair with this for a few >>>> days now and would love any insight into it. >>>> >>>> >>>> Thanks in advance, >>>> >>>> Sukhmeet >>>> >>>> >>>> p.s. Older threads on this topic, which no solution that worked for me: >>>> >>>> https://groups.google.com/forum/#!newtopic/web2py/web2py/hmsupVHdDHo >>>> >>>> http://stackoverflow.com/questions/5539722/standalone-web2py-dal-with-pyramid-requests-threads-and-memory >>>> >>>> >>>> -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
