[issue41588] Potential Memory leak with concurrent.futures.ThreadPoolExecutor's map

[Kyle Stanley]

Change by Kyle Stanley :


--
nosy: +aeros, bquinlan, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41588] Potential Memory leak with concurrent.futures.ThreadPoolExecutor's map

[Or Yahalom]

New submission from Or Yahalom :

I've been debugging a high memory consumption in one of my scripts and traced 
it back to the `concurrent.futures.ThreadPoolExecutor`.

When further investigating and playing around, I found out that when using 
`concurrent.futures.ThreadPoolExecutor` with the map function, and passing a 
dictionary to the map's function as an argument, the memory used by the pool 
won't be freed and as a result the total memory consumption will continue to 
rise. (Seems like it also happens when passing a list and maybe even other 
types).

Here is an example of a code to recreate this issue:

```
#!/usr/bin/env python3

import os
import time
import psutil
import random
import concurrent.futures

from memory_profiler import profile as mem_profile

p = psutil.Process(os.getpid())

def do_magic(values):
return None

@mem_profile
def foo():
a = {i: chr(i) for i in range(1024)}
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
proccessed_data = pool.map(do_magic, a)

def fooer():
while True:
foo()
time.sleep(1)

fooer()
```

--
components: Extension Modules
messages: 375647
nosy: or12
priority: normal
severity: normal
status: open
title: Potential Memory leak with concurrent.futures.ThreadPoolExecutor's map
type: resource usage
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com