New submission from Lorenz Huedepohl <lor...@mpa-garching.mpg.de>: The UNIX mmap version in
Modules/mmapmodule.c makes a call to dup() to duplicate the file descriptor that is passed for creating the memory-mapped region. This way, I frequently hit the limit for the number of open file handles while creating mmap.mmap() instances, despite closing all my opened files after creating a mapping for them. My application is scientific data (read: "large data" :-) analysis for which mmap() is very well suited together with numpy/scipy - however, the large number of files causes me to hit the resource limit on open file handles) I propose to remove this dup(), which was apparently introduced in the process of fixing issue #728515, concerning incorrect behavior of the mmap.resize() method on UNIX, as it was feared the user could have closed the file handle already. I think it should be the responsibility of the user not to close the file in question or either - if it needs to be closed - not to call the resize method later. I should stress that a call to mmap(2) does not require an open file handle during the time of the mapping but that a duplicate of the file handle was only kept to allow .size() and .resize() to work. See the POSIX Programmer's Manual: The mmap() function shall add an extra reference to the file associated with the file descriptor fildes which is not removed by a subsequent close() on that file descriptor. This reference shall be removed when there are no more mappings to the file. It would be great if this little nicety would translate better to Python! Regards, Lorenz ---------- components: Library (Lib) files: no_dup.patch keywords: patch messages: 126104 nosy: lorenz priority: normal severity: normal status: open title: UNIX mmap unnecessarily dup() file descriptor type: resource usage versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3 Added file: http://bugs.python.org/file20373/no_dup.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue10897> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com