[issue43911] Queue.get() memory leak

Jens added the comment: Raymond, thanks for your suggestions. My deployed applications don't hold up 20m items at a time, that was a way to show the leak. I was able to resolve the threading, queue-based leaks on my instances by modifying the Queue, Event and Conditions classes to use

[issue43911] Queue.get() memory leak

Raymond Hettinger added the comment: Here's a more compact version of the variant with an underlying dict: class MyQueue(Queue): def _init(self, maxsize): self.end = 0 self.q = {} def _qsize(self): return len(self.q) def

[issue43911] Queue.get() memory leak

Raymond Hettinger added the comment: > I don't think this is necessarily specific to my local build Replace "local" with "specific to a particular combination of C compiler and operating system". On my Mac, the effect mostly doesn't occur as all, 0.05% before the run and 0.10% after the

[issue43911] Queue.get() memory leak

Jens added the comment: Ok, I see, thanks Raymond. Queue based logging leaks seem to hang my deployed applications atm, so this seemed like a possible reason for it. I use locally 8GB Ubuntu, and it gets to 2.2% after return with 20million puts, and on a remote 1GB Ubuntu instance, it

[issue43911] Queue.get() memory leak

Change by Raymond Hettinger : -- assignee: -> rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue43911] Queue.get() memory leak

Raymond Hettinger added the comment: FWIW, when running your code on my Intel Mac with 16 Gb of RAM, the "after return" result is 0.1%. So getting up to 2.2% seems to be specific to your build. Also, I just ran your code with a deque instrumented to count the number of mallocs and frees.

[issue43911] Queue.get() memory leak

Jens added the comment: Also compared this library to deque, and Queues based on this: https://github.com/kata198/python-cllist It seems to be as fast as deque, uses a bit more memory at the top usage, but does not leak at all. -- ___ Python

[issue43911] Queue.get() memory leak

Jens added the comment: So this got me thinking of trying to use some other linked list implementations. I've used a llist library - https://github.com/ajakubek/python-llist Using their doubly linked list implementation: class DllistQueue(queue.Queue): def _init(self,

[issue43911] Queue.get() memory leak

Jens added the comment: Regarding deque, the leak indeed does not seem to be releasable after it is inited to up the size of the number of elements that are going to put into the queue, as: qmem = collections.deque(range(n_puts)) qmem.clear() The results are: >#

[issue43911] Queue.get() memory leak

Jens added the comment: Thanks for your input. So i've run the tests with the List of Lists Queue class, there seems to be a resulting difference depending on what qsize() method I define, that is called my script. For an instance where qsize just return None, class

[issue43911] Queue.get() memory leak

Raymond Hettinger added the comment: Also, when running deque measurements, run the following before getting the starting process memory: deque(range(2000)).clear() # fill the deque freelist That will give a cleaner before/after comparison. --

[issue43911] Queue.get() memory leak

Raymond Hettinger added the comment: For a large amount of data, a list uses a single large contiguous block of memory while a deque uses many small discontiguous blocks. In your demo, I that suspect that some of the memory pages for deque's blocks are also being used for other small bits

[issue43911] Queue.get() memory leak

Jens added the comment: Results for queue._PySimpleQueue: ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 37.61% >-- gets done - qsize 0 >mem_pct 2.22% >deleting queue after gets

[issue43911] Queue.get() memory leak

Jens added the comment: Just inspected the PriorityQueue and LifoQueue classes, they dont employ a deque at all but simply a list, but all other Queues tested do (except the native SimpleQueue). Since they don't leak, the leak itself seems to be coming from deque, and the fact that it does

[issue43911] Queue.get() memory leak

Jens added the comment: Hi, Thanks for your reply, so I've run same script with queue.PriorityQueue, queue.LifoQueue, queue.SimpleQueue, Asyncio.Queue as well as collections.dequeue 1. PriorityQueue ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct

[issue43911] Queue.get() memory leak

Raymond Hettinger added the comment: This may be a memory fragmentation problem and likely doesn't have anything to do with Queue instances. As an experiment, try using queue.PriorityQueue and queue.LifoQueue to see if the issue persists. -- nosy: +rhettinger

[issue43911] Queue.get() memory leak

Jens added the comment: I've tried to profile the memory with tracemalloc as well as pympler, but they dont show any leaks at python level, so I suspect that might be a C level leak. -- ___ Python tracker

[issue43911] Queue.get() memory leak

New submission from Jens : I use the following code to produce what looks like a memory leak after emptying a queue with the get() method. import queue import os import psutil def run(del_after_puts, del_after_gets, n_puts, process): mem = queue.Queue()