[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Yongzhi Pan added the comment: On macOS with Python 3.7.2, using pitrou's code, I suspect Python does not delete some semaphores used by Queue. Run these: import multiprocessing import os import threading os.system('lsof -p {} | grep -v txt'.format(os.getpid())) q =

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Change by Yongzhi Pan : -- nosy: +fossilet ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Chris Langton added the comment: interestingly, while it is expected Process or Queue would actually close resource file descriptors and doesn't because a dev decided they prefer to defer to the user how to manage gc themselves, the interesting thing is if you 'upgrade' your code to use a

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Chris Langton added the comment: @pitrou I am interested in a fix for Python 2.7 because in Python 3.x the manner in which arithmetic is output is not arbitrary precise. So I will continue using Python 2.7 until another language I am familiar with that has superior arbitrary precise

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Antoine Pitrou added the comment: Thanks for reporting this. I agree this is a real issue, but it doesn't exist on Python 3 anymore: >>> q = multiprocessing.Queue() >>> q.put(1) >>> q.get() 1 >>> threading.enumerate() [<_MainThread(MainThread, started 139978753529600)>,

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: The garbage collector in Python does not work like that. If an object reaches zero references is destroyed immediately. The only problem is when circular references exist and is in this case when object deletion is delayed until

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: You're a missing the fact that in C++, there is no garbage collector. The destructor both releases resources and deallocates memory. There is no window between releasing resources and object destruction. Python, while not exactly like

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: > RAII in C++ ensures that, on object destruction, resources that have been > acquired will be closed and deallocated. Which is exactly what is happening here. When the queue gets destroyed (because the reference count reaches 0

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: Your comparison is not correct. RAII in C++ ensures that, on object destruction, resources that have been acquired will be closed and deallocated. The closest analogy in Python is the use of a context manager, which, btw, a Queue does

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: >"I want to terminate the queue and make sure that no resources are leaked. Then you don't need to do anything special, those will be cleared on object destruction. This is not an unusual pattern even in other languages. For

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: I don't want to "close the pipes but maintain the queue alive" - I want to terminate the queue and make sure that no resources are leaked. It's that simple. When one closes a file or a socket, there is no underlying OS resource being

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: Notice that the documentation for close says: > Indicate that no more data will be put on this queue by the current process. > The background thread will quit once it has flushed all buffered data to the > pipe. This is called

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: @pablo: I am using Python 2.7.12 (distributed with Ubuntu 16), what are you using? This might explain the difference between what we see. Yet, irrespective of this difference, imho, it would be a better design to have "close" actually

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: When I run your script I do not see any file descriptor associated with the queue when subprocess.call(["ls", "-la", "/proc/%d/fd" % os.getpid()]) is executed. This is my output if I just execute your program: starting 3728

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: Here is the repro (I am running this on Ubuntu 16 with the stock Python version 2.7.12): #!/usr/bin/env python import os

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Change by Ned Deily : -- nosy: +davin, pitrou ___ Python tracker ___ ___ Python-bugs-list

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: Unfortunately this is not the case. I will shrink my repro down to a more manageable size and post it here. -- ___ Python tracker

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: Notice that the writer gets closed when it receives a sentinel value (which how the queue knows when to close as part of the design): >>> x.put(multiprocessing.queues._sentinel) If you call close after this line there will not be

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: The way is to delete the object. IMHO I would not say is "leaked" as the object is still alive and holds resources and these resources are properly handled on destruction. I cannot think of an immediate use case of closing both

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: Pablo, but there is no way to close the other side. Indeed, if you look in the implementation, you will see that the writer file descriptor can't be closed. -- ___ Python tracker

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Pablo Galindo Salgado added the comment: Is not leaked as the object is still alive. Only one side is closed when you call "closed". If you destroy the object: >>> del x You will see that there are no more file descriptors in /proc/PID/fd associated with that queue.

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

Henrique Andrade added the comment: Correcting my original - after close(): > ll /proc/8096/fd total 0 dr-x-- 2 hcma hcma 0 2018-03-15 14:03:23.210089578 -0400 . dr-xr-xr-x 9 hcma hcma 0 2018-03-15 14:03:23.190089760 -0400 .. lrwx-- 1 hcma hcma 64 2018-03-15

[issue33081] multiprocessing Queue leaks a file descriptor associated with the pipe writer

New submission from Henrique Andrade : A simple example like such demonstrates that one of the file descriptors associated with the underlying pipe will be leaked: >>> from multiprocessing.queues import Queue >>> x = Queue() >>> x.close() Right after the queue is created