New submission from Jonas Brunsgaard:
When making repeated calls to queue.get, memory is building up and is not freed
until queue.push is called.
I wrote this little program to show my findings. The program will perform a lot
of calls to queue.get and once every 60 seconds a queue.push is performed.
Every 15 seconds the memory usage of dictionaries is printet to the console.
You can find the output below the program
```
import asyncio
from pympler import muppy
from pympler import summary
q = asyncio.Queue()
loop = asyncio.get_event_loop()
closing = False
async def get_with_timeout():
while not closing:
try:
task = asyncio.ensure_future(q.get())
await asyncio.wait_for(task, 0.2)
except asyncio.TimeoutError:
pass
def mem_profiling():
if not closing:
types_ = muppy.filter(muppy.get_objects(), Type=dict)
summary.print_(summary.summarize(types_))
loop.call_later(15, mem_profiling)
def put():
q.put_nowait(None)
loop.call_later(60, put)
put()
tasks = [asyncio.ensure_future(get_with_timeout()) for _ in range(10000)]
mem_profiling()
try:
loop.run_forever()
except KeyboardInterrupt:
closing = True
loop.run_until_complete(
asyncio.ensure_future(asyncio.wait(tasks)))
finally:
loop.close()
```
Output:
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 11886 | 3.20 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 224835 | 62.44 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 401487 | 111.00 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 576621 | 158.86 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 219745 | 58.99 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 390061 | 105.72 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 560623 | 152.65 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
<class 'dict | 220372 | 59.11 MB
<class 'collections.OrderedDict | 11 | 16.84 KB
<class 'email._encoded_words._QByteMap | 1 | 288 B
types | # objects | total size
======================================== | =========== | ============
What we see is that memory builds up ~3mb/s, and when the push method is called
the memory usage returns to normal.
Is this the expected behavior or is this a bug? If it is expected I think we
should update the documentation, to let people know about this behavior.
--
Jonas Brunsgaard
----------
components: asyncio
files: poc.py
messages: 259327
nosy: Jonas Brunsgaard, gvanrossum, haypo, yselivanov
priority: normal
severity: normal
status: open
title: Memleak when repeated calls to asyncio.queue.Queue.get is performed,
without push to queue.
type: resource usage
versions: Python 3.5
Added file: http://bugs.python.org/file41772/poc.py
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26259>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
