New submission from Jens <multiks2...@gmail.com>:
I use the following code to produce what looks like a memory leak after
emptying a queue with the get() method.
import queue
import os
import psutil
def run(del_after_puts, del_after_gets, n_puts, process):
mem = queue.Queue()
for msg in range(n_puts):
msg_put =
f'{msg}_0000000000000000000000000000000000000000000000000000000000000333333333333331111111111'
if msg % 1000000 == 0:
print(f'puting {msg} qsize {len(mem.queue)}')
mem.put(msg_put)
print(f'------ put done ----- qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_puts:
print(f'deleting queue after puts {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
for _ in range(n_puts):
msg_get = mem.get()
msg = int(msg_get.split('_')[0])
if msg % 1000000 == 0:
print(f'getting_q {msg} qsize {len(mem.queue)} ')
mem.task_done()
print(f'------ gets done ----- qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_gets:
print(f'deleting queue after gets {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
if __name__ == '__main__':
del_after_puts = False
del_after_gets = False
n_puts = 20_000_000
print()
print('#########')
print(f'del_after_puts {del_after_puts} del_after_gets
{del_after_gets} n_puts {n_puts}')
process = psutil.Process(os.getpid())
print('before run')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
run(del_after_puts, del_after_gets, n_puts, process)
print(f'after run return')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
This script can be run in 3 ways:
1. Add n_puts elements into the queue and then empty it.
2. Add n_puts elements into the queue and then delete the queue object
3. Add n_puts elements into the queue and then empty it and then delete the
queue object.
For the 1st and 3rd case, the script seems to produce a memory leak as in the
following:
1st case, before putting elements into the queue mem used is 0.15%, after
emptying it 2.22%:
> #########
>
> del_after_puts False del_after_gets False n_puts 20000000
>
> before run
>
> mem_pct 0.15%
>
> ------ put done ----- qsize 20000000
>
> mem_pct 37.61%
>
> ------ gets done ----- qsize 0
>
> mem_pct 2.22%
3rd case, before putting elements into the queue mem used is 0.15%, after
emptying it 2.22%, after deleting the object, 2.22%:
> #########
>
> del_after_puts False del_after_gets True n_puts 20000000
>
> before run
>
> mem_pct 0.15%
>
> ------ put done ----- qsize 20000000
>
> mem_pct 37.61%
>
> ------ gets done ----- qsize 0
>
> mem_pct 2.22%
>
> deleting queue after gets <queue.Queue object at 0x7fbd87295a10>
>
> mem_pct 2.22%
For the 2nd case, mem_pct at the start is 0.15%, after putting all elements
into the queue and just deleting it, 0.16%, which is almost the same.
> #########
>
> del_after_puts True del_after_gets False n_puts 20000000
>
> before run
>
> mem_pct 0.15%
>
> ------ put done ----- qsize 20000000
>
> mem_pct 37.61%
>
> deleting queue after puts <queue.Queue object at 0x7f29084eca10>
>
> mem_pct 0.16%
As it can be seen, memory returns to the start level only in the first case
when only `queue.put()` is invoked, hence it seems that `queue.get()` produces
a memory leak.
This is persistent across python 3.7, 3.8, as well as 3.9.
----------
messages: 391586
nosy: multiks2200
priority: normal
severity: normal
status: open
title: Queue.get() memory leak
versions: Python 3.9
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue43911>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
