Hi,
first of all, sorry for being such a pest, but all former attempts to solve
this issue on other fora has been grinding to a halt.
In short: I try to combine a context manager with ctypes structures on memory
mapped files in order to manage huge binary files. (An approach, that performs
great, while easy to use and keeping the resource usage low).
FWIW, the code is targeted for Linux environments running Python3.
The smallest script demonstrating the issue (thanks to Peter Otten):
import ctypes
import mmap
from contextlib import contextmanager
class T(ctypes.Structure):
_fields = [("foo", ctypes.c_uint32)]
@contextmanager
def map_struct(m, n):
m.resize(n * mmap.PAGESIZE)
yield T.from_buffer(m)
SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
with map_struct(m, 1) as a:
a.foo = 1
with map_struct(m, 2) as b:
b.foo = 2
resulting in:
$ python3 mmap_test.py
Traceback (most recent call last):
File "mmap_test.py", line 23, in <module>
with map_struct(m, 2) as b:
File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
return next(self.gen)
File "mmap_test.py", line 12, in map_struct
m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.
Python2 does not crash, but that's a different story. What happens here is:
the context manager variable "a" keeps a reference to a memory mapped area
alive, that results in a unresizable and not properly closable mmap.
Right now, this rather ugly and error prone workaround must be used, that
renders the purpose of the context manager ad absurdum:
with map_struct(m, 1) as a:
a.foo = 1
del a
with map_struct(m, 2) as b:
b.foo = 2
del b
In order to get this working properly, the ctypes mapping needs a method to
free the mapping actively. E.g.:
@contextmanager
def map_struct(m, n):
m.resize(n * mmap.PAGESIZE)
yield T.from_buffer(m)
T.unmap_buffer(m)
Other attempts with weakref and the like do not work due to the nature of the
ctypes types.
My own investigations in the _ctypes module were unsuccessful so far.
Hopefully, somebody in the audience cares enough for this module in order to
get this fixed up (or probably I'm missing something obvious..).
Any ideas are very much appreciated.
Pete
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
