[issue38430] Memory leak in ThreadPoolExecutor + run_in_executor

[Evgeny Nizhibitsky]

Evgeny Nizhibitsky  added the comment:

Oh, I see that in the initial code with leakage (it was heavy 
ThreadPoolExecutor + xgboost thing) there was an await but I must have lost it 
somewhere while reducing it to the minimal example and finished in the wrong 
direction.

Glad too see that it raised a discussion to prevent others from getting into 
this silent trap.

--

___
Python tracker 
<https://bugs.python.org/issue38430>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38430] Memory leak in ThreadPoolExecutor + run_in_executor

[Evgeny Nizhibitsky]

New submission from Evgeny Nizhibitsky :

I have run into a memory leak caused by using run_in_executor + 
ThreadPoolExecutor while running some stability tests with custom web services.

It was 1 MB leaked for 1k requests made for my case and I've extracted the root 
cause and converted it into minimal script with both mentioned parts + just NOP 
function to "run".

The script can easily eat up to 1 GB of memory in less then 1 minute now. It 
uses external psutil library to report the memory allocated but it can be 
easily commented out and the leak will stay anyway.

One can found that script attached + Dockerfile/Makefile for reproducibility. 
I've also reproduced it in my own conda-based 3.7 environment as well as the 
master branch of cpython.

--
components: asyncio
files: python-leak.zip
messages: 354351
nosy: Evgeny Nizhibitsky, asvetlov, yselivanov
priority: normal
severity: normal
status: open
title: Memory leak in ThreadPoolExecutor + run_in_executor
type: resource usage
versions: Python 3.6, Python 3.7, Python 3.9
Added file: https://bugs.python.org/file48654/python-leak.zip

___
Python tracker 
<https://bugs.python.org/issue38430>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com