[Python-Dev] pyparallel and new memory API discussions...
The new memory API discussions (and PEP) warrant a quick pyparallel
update: a couple of weeks after PyCon, I came up with a solution for
the biggest show-stopper that has been plaguing pyparallel since its
inception: being able to detect the modification of "main thread"
Python objects from within a parallel context.
For example, `data.append(4)` in the example below will generate an
AssignmentError exception, because data is a main thread object, and
`data.append(4)` gets executed from within a parallel context::
data = [ 1, 2, 3 ]
def work():
data.append(4)
async.submit_work(work)
The solution turned out to be deceptively simple:
1. Prior to running parallel threads, lock all "main thread"
memory pages as read-only (via VirtualProtect on Windows,
mprotect on POSIX).
2. Detect attempts to write to main thread pages during parallel
thread execution (via SEH on Windows or a SIGSEGV trap on POSIX),
and raise an exception instead (detection is done in the ceval
frame exec loop).
3. Prior to returning control back to the main thread (which will
be paused whilst all the parallel threads are running), unlock
all the "main thread" pages.
4. Pause all parallel threads while the main thread runs.
5. Go back to 1.
I got a proof-of-concept working on Windows a while back (and also
played around with large page support in the same commit). The main
changes were to obmalloc.c:
https://bitbucket.org/tpn/pyparallel/commits/0e70a0caa1c07dc0c14bb5c99cbe808c1c11779f#chg-Objects/obmalloc.c
The key was the introduction of two new API calls, intended to be
called by the pyparallel.c infrastructure:
_PyMem_LockMainThreadPages()
_PyMem_UnlockMainThreadPages()
The implementation is pretty simple:
+int
+_PyMem_LockMainThreadPages(void)
+{
+DWORD old = 0;
+
+if (!VirtualProtect(base_addr, nbytes_committed, PAGE_READONLY, &old)) {
+PyErr_SetFromWindowsErr(0);
+return -1;
+}
Note the `base_addr` and `nbytes_committed` argument. Basically, I
re-organized obmalloc.c a little bit such that we never actually
call malloc() directly. Instead, we exploit the ability to reserve
huge virtual address ranges without actually committing the memory,
giving us a fixed `base_addr` void pointer that we can pass to calls
like VirtualProtect or mprotect.
We then incrementally commit more pages as demand increases, and
simply adjust our `nbytes_committed` counter as we go along. The
net effect is that we can call VirtualProtect/mprotect once, with a
single base void pointer and size_t range, and immediately affect the
protection of all memory pages that fall within that range.
As an added bonus, we also get a very cheap and elegant way to test
if a pointer (or any arbitrary memory address, actually) belongs to
the main thread's memory range (at least in comparison to the
existing _PyMem_InRange black magic). (This is very useful for my
pyparallel infrastructure, which makes extensive use of conditional
logic based on address tests.)
(Side-bar: a side-effect of the approach I've used in the proof-
of-concept (by only having a single base addr pointer) is that
we effectively limit the maximum memory we could eventually
commit.
I actually quite like this -- in fact, I'd like to tweak it
such that we can actually expose min/max memory values to the
Python interpreter at startup (analogous to the JVM).
Having known upper bounds on maximum memory usage will vastly
simplify some other areas of my pyparallel work (like the async
socket stuff).
For example, consider network programs these days that take a
"max clients" configuration parameter. That seems a bit
backwards to me.
It would be better if we simply said, "here, Python, you have
1GB to work with". That allows us to calculate how many
clients we could simultaneously serve based on socket memory
requirements, which allows for much more graceful behavior
under load than leaving it open-ended.
Maximum memory constraints would also be useful for the
parallel.map(callable, iterable) stuff I've got in the works,
as it'll allow us to optimally chunk work and assign to threads
based on available memory.)
So, Victor, I'm interested to hear how the new API you're proposing
will affect this solution I've come up with for pyparallel; I'm
going to be absolutely dependent upon the ability to lock main
thread pages as read-only in one fell-swoop -- am I still going to
be able to do that with your new API in place?
Regards,
Trent.
Re: [Python-Dev] pyparallel and new memory API discussions...
On 19 June 2013 23:10, Trent Nelson wrote: > So, Victor, I'm interested to hear how the new API you're proposing > will affect this solution I've come up with for pyparallel; I'm > going to be absolutely dependent upon the ability to lock main > thread pages as read-only in one fell-swoop -- am I still going to > be able to do that with your new API in place? By default, nothing will change for the ordinary CPython runtime. It's only if an embedding application starts messing with the allocators that things might change, but at that point, pyparallel would break anyway. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
2013/6/19 Trent Nelson : > > The new memory API discussions (and PEP) warrant a quick pyparallel > update: a couple of weeks after PyCon, I came up with a solution for > the biggest show-stopper that has been plaguing pyparallel since its > inception: being able to detect the modification of "main thread" > Python objects from within a parallel context. > > For example, `data.append(4)` in the example below will generate an > AssignmentError exception, because data is a main thread object, and > `data.append(4)` gets executed from within a parallel context:: > > data = [ 1, 2, 3 ] > > def work(): > data.append(4) > > async.submit_work(work) > > The solution turned out to be deceptively simple: > > 1. Prior to running parallel threads, lock all "main thread" > memory pages as read-only (via VirtualProtect on Windows, > mprotect on POSIX). > > 2. Detect attempts to write to main thread pages during parallel > thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), > and raise an exception instead (detection is done in the ceval > frame exec loop). Quick stupid question: because of refcounts, the pages will be written to even in case of read-only access. How do you deal with this? cf ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
Le Tue, 18 Jun 2013 22:40:49 +0200, Victor Stinner a écrit : > > Other changes > - > [...] > > * Configure external libraries like zlib or OpenSSL to allocate memory > using ``PyMem_RawMalloc()`` Why so, and is it done by default? > Only one get/set function for block allocators > -- > > Replace the 6 functions: > > * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` > * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` > > with 2 functions with an additional *domain* argument: > > * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator > *allocator)`` > * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator > *allocator)`` I would much prefer this solution. > Drawback: the caller has to check if the result is 0, or handle the > error. Or you can just call Py_FatalError() if the domain is invalid. > If an hook is used to the track memory usage, the ``malloc()`` memory > will not be seen. Remaining ``malloc()`` may allocate a lot of memory > and so would be missed in reports. A lot of memory? In main()? Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
Hi Charles-François! Good to hear from you again. It was actually your e-mail a few months ago that acted as the initial catalyst for this memory protection idea, so, thanks for that :-) Answer below. On Wed, Jun 19, 2013 at 07:01:49AM -0700, Charles-François Natali wrote: > 2013/6/19 Trent Nelson : > > > > The new memory API discussions (and PEP) warrant a quick pyparallel > > update: a couple of weeks after PyCon, I came up with a solution for > > the biggest show-stopper that has been plaguing pyparallel since its > > inception: being able to detect the modification of "main thread" > > Python objects from within a parallel context. > > > > For example, `data.append(4)` in the example below will generate an > > AssignmentError exception, because data is a main thread object, and > > `data.append(4)` gets executed from within a parallel context:: > > > > data = [ 1, 2, 3 ] > > > > def work(): > > data.append(4) > > > > async.submit_work(work) > > > > The solution turned out to be deceptively simple: > > > > 1. Prior to running parallel threads, lock all "main thread" > > memory pages as read-only (via VirtualProtect on Windows, > > mprotect on POSIX). > > > > 2. Detect attempts to write to main thread pages during parallel > > thread execution (via SEH on Windows or a SIGSEGV trap on POSIX), > > and raise an exception instead (detection is done in the ceval > > frame exec loop). > > Quick stupid question: because of refcounts, the pages will be written > to even in case of read-only access. How do you deal with this? Easy: I don't refcount in parallel contexts :-) There's no need, for two reasons: 1. All memory allocated in a parallel context is localized to a private heap. When the parallel context is finished, the entire heap can be blown away in one fell-swoop. There's no need for reference counting or GC because none of the objects will exist after the parallel context completes. 2. The main thread won't be running when parallel threads/contexts are executing, which means main thread objects being accessed in parallel contexts (read-only access is fine) won't be suddenly free()'d or GC-collected or whatever. You get credit for that second point; you asked a similar question a few months ago that made me realize I absolutely couldn't have the main thread running at the same time the parallel threads were running. Once I accepted that as a design constraint, everything else came together nicely... "Hmmm, if the main thread isn't running, it won't need write-access to any of its pages! If we mark them read-only, we could catch the traps/SEHs from parallel threads, then raise an exception, ahh, simple!". I'm both chuffed at how simple it is (considering it was *the* major show-stopper), and miffed at how it managed to elude me for so long ;-) Regards, Trent. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
2013/6/19 Antoine Pitrou : > Le Tue, 18 Jun 2013 22:40:49 +0200, > Victor Stinner a écrit : >> >> Other changes >> - >> > [...] >> >> * Configure external libraries like zlib or OpenSSL to allocate memory >> using ``PyMem_RawMalloc()`` > > Why so, and is it done by default? (Oh, I realized that PyMem_Malloc() may be used instead of PyMem_RawMalloc() if we are sure that the library will only be used when the GIL is held.) "is it done by default?" First, it would be safer to only reuse PyMem_RawMalloc() allocator if PyMem_SetRawMalloc() was called. Just to avoid regressions in Python 3.4. Then, it depends on the library: if the allocator can be replaced for one library object (ex: expat supports this), it can always be replaced. Otherwise, we should only replace the library allocator if Python is a standalone program (don't replace the library allocator if Python is embedded). That's why I asked if it is possible to check if Python is embedded or not. "Why so," For the "track memory usage" use case, it is important to track memory allocated in external libraries to have accurate reports, because these allocations may be huge. >> Only one get/set function for block allocators >> -- >> >> Replace the 6 functions: >> >> * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` >> * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` >> >> with 2 functions with an additional *domain* argument: >> >> * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator >> *allocator)`` >> * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator >> *allocator)`` > > I would much prefer this solution. I don't have a strong preference between these two choices. Oh, one argument in favor of one generic function is that code using these functions would be simpler. Extract of the unit test of the implementation (_testcapi.c): +if (api == 'o') +PyObject_SetAllocator(&hook.alloc); +else if (api == 'r') +PyMem_SetRawAllocator(&hook.alloc); +else +PyMem_SetAllocator(&hook.alloc); With a generic function, this block can be replace with one unique function call. >> Drawback: the caller has to check if the result is 0, or handle the >> error. > > Or you can just call Py_FatalError() if the domain is invalid. I don't like Py_FatalError(), especially when Python is embedded. It's safer to return -1 and expect the caller to check for the error case. >> If an hook is used to the track memory usage, the ``malloc()`` memory >> will not be seen. Remaining ``malloc()`` may allocate a lot of memory >> and so would be missed in reports. > > A lot of memory? In main()? Not in main(). The Python expat and zlib modules call directly malloc() and may allocate large blocks. External libraries like OpenSSL or bz2 may also allocate large blocks. See issues #18203 and #18227. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
Right, think of the "ctxt" as a "this" pointer from c++. If you have an allocator object, that you got from some c++ api, and want to ask Python to use that, you need to be able to thunk the "this" pointer to get at the particular allocator instance. It used to be a common mistake when writing C callback apis to forget to add an opaque "context" pointer along with the callback function. This omission makes it difficult (but not impossible) to attach c++ methods to such callbacks. K > -Original Message- > From: Python-Dev [mailto:python-dev- > [email protected]] On Behalf Of Scott Dial > Sent: 19. júní 2013 04:34 > To: [email protected] > Cc: [email protected] > Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python > memory allocators > > On 6/18/2013 11:32 PM, Nick Coghlan wrote: > > Agreed more of that rationale needs to be moved from the issue tracker > > into the PEP, though. > > Thanks for the clarification. I hadn't read the issue tracker at all. On it's > face > value, I didn't see what purpose it served, but having read Kristján's > comments on the issue tracker, he would like to store state for the allocators > in that ctx pointer.[1] Having read that (previously, I thought the only > utility > was distinguishing which domain it was -- a small, glorified enumeration), but > his use-case makes sense and definitely is informative to have in the PEP, > because the utility of that wasn't obvious to me. > > Thanks, > -Scott > > [1] http://bugs.python.org/issue3329#msg190529 > """ > One particular trick we have been using, which might be of interest, is to be > able to tag each allocation with a "context" id. This is then set according > to a > global sys.memcontext variable, which the program will modify according to > what it is doing. This can then be used to track memory usage by different > parts of the program. > """ > > -- > Scott Dial > [email protected] > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/kristjan%40ccpgames.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On Wed, 19 Jun 2013 17:24:21 +0200 Victor Stinner wrote: > > For the "track memory usage" use case, it is important to track memory > allocated in external libraries to have accurate reports, because > these allocations may be huge. [...] > Not in main(). The Python expat and zlib modules call directly > malloc() and may allocate large blocks. External libraries like > OpenSSL or bz2 may also allocate large blocks. Fair enough. > >> Drawback: the caller has to check if the result is 0, or handle the > >> error. > > > > Or you can just call Py_FatalError() if the domain is invalid. > > I don't like Py_FatalError(), especially when Python is embedded. It's > safer to return -1 and expect the caller to check for the error case. I don't think you need to check for errors. The domain is always one of the existing constants, i.e. it should be hard-coded in the source, not computed. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
2013/6/19 Antoine Pitrou : > On Wed, 19 Jun 2013 17:24:21 +0200 >> >> Drawback: the caller has to check if the result is 0, or handle the >> >> error. >> > >> > Or you can just call Py_FatalError() if the domain is invalid. >> >> I don't like Py_FatalError(), especially when Python is embedded. It's >> safer to return -1 and expect the caller to check for the error case. > > I don't think you need to check for errors. The domain is always one of > the existing constants, i.e. it should be hard-coded in the source, not > computed. Imagine that PyMem_GetBlockAllocator() is part of the stable ABI and that a new domain is added to Python 3.5. An application is written for Python 3.5 and is run with Python 3.4: how would the application notice that PyMem_GetBlockAllocator() does not know the new domain? "I don't think you need to check for errors." Do you mean that an unknown domain should be simply ignored? Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
> 1. All memory allocated in a parallel context is localized to a > private heap. How do you allocate memory in this "private" heap? Did you add new functions to allocate memory? Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
Oh, it should be public, in my opinion. We do exactly that when we embed python into UnrealEngine. We keep pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas. That way, we can still keep track of python's memory usage from with the larger application, even if the granularity of memory is now on an "arena" level, rather than individual allocs. K > -Original Message- > From: Python-Dev [mailto:python-dev- > [email protected]] On Behalf Of Victor Stinner > Sent: 18. júní 2013 21:20 > To: Python Dev > Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python > memory allocators > > typedef struct { > /* user context passed as the first argument >to the 2 functions */ > void *ctx; > > /* allocate a memory mapping */ > void* (*alloc) (void *ctx, size_t size); > > /* release a memory mapping */ > void (*free) (void *ctx, void *ptr, size_t size); > } PyMemMappingAllocator; > > The PyMemMappingAllocator structure is very specific to the pymalloc > allocator. There is no "resize", "lock" nor "protect" method. There is no way > to configure protection or flags of the mapping. The > PyMem_SetMappingAllocator() function was initially called > _PyObject_SetArenaAllocator(). I'm not sure that the structure and the > 2 related functions should be public. Can an extension module call private > (_Py*) functions or use a private structure? > > Or the structure might be renamed to indicate that it is specific to arenas? > > What do you think? > > Victor > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/kristjan%40ccpgames.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote: > > 1. All memory allocated in a parallel context is localized to a > > private heap. > > How do you allocate memory in this "private" heap? Did you add new > functions to allocate memory? Yup: _PyHeap_Malloc(): http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365. All memory operations (PyObject_New/Malloc etc) get intercepted during parallel thread execution and redirected to _PyHeap_Malloc(), which is a very simple slab allocator. (No need for convoluted buckets because we never free individual objects during parallel execution; instead, we just blow everything away at the end.) Trent. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
2013/6/19 Kristján Valur Jónsson : > Oh, it should be public, in my opinion. Ok. And do you think that the PyMemMappingAllocator structure is complete, or that we should add something to be future-proof? At least, PyMemMappingAllocator is enough for pymalloc usage :-) Is PyMemMappingAllocator complete enough for your usage at CCP Games? > We do exactly that when we embed python into UnrealEngine. We keep pythons > internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas. > That way, we can still keep track of python's memory usage from with the > larger application, even if the granularity of memory is now on an "arena" > level, rather than individual allocs. I hope that the PEP 445 is flexible enough to allow you to decide which functions are hooked and replaced, and which functions will be leaved unchanged. That's why I'm not in favor of the "Make PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On 6/19/2013 11:24 AM, Victor Stinner wrote: 2013/6/19 Antoine Pitrou : Le Tue, 18 Jun 2013 22:40:49 +0200, Victor Stinner a écrit : Only one get/set function for block allocators -- Replace the 6 functions: * ``void PyMem_GetRawAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyObject_GetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_SetRawAllocator(PyMemBlockAllocator *allocator)`` * ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)`` * ``void PyObject_SetAllocator(PyMemBlockAllocator *allocator)`` with 2 functions with an additional *domain* argument: * ``int PyMem_GetBlockAllocator(int domain, PyMemBlockAllocator *allocator)`` * ``int PyMem_SetBlockAllocator(int domain, PyMemBlockAllocator *allocator)`` I would much prefer this solution. I do to. The two names can be remembered as one pair with only get/set difference. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
"""
So, Victor, I'm interested to hear how the new API you're proposing
will affect this solution I've come up with for pyparallel; I'm
going to be absolutely dependent upon the ability to lock main
thread pages as read-only in one fell-swoop -- am I still going to
be able to do that with your new API in place?
"""
2013/6/19 Trent Nelson :
> On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
>> > 1. All memory allocated in a parallel context is localized to a
>> > private heap.
>>
>> How do you allocate memory in this "private" heap? Did you add new
>> functions to allocate memory?
>
> Yup:
> _PyHeap_Malloc():
> http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
>
> All memory operations (PyObject_New/Malloc etc) get intercepted
> during parallel thread execution and redirected to _PyHeap_Malloc(),
> which is a very simple slab allocator. (No need for convoluted
> buckets because we never free individual objects during parallel
> execution; instead, we just blow everything away at the end.)
Ok, so I don't think that the PEP 445 would change anything for you.
The following change might have an impact: If _PyHeap_Malloc is not
thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
GIL is not held would avoid bugs in your code.
If you want to choose dynamically the allocator at runtime, you can
replace PyObject_Malloc allocator using:
-- 8< -
static void *
_PxMem_AllocMalloc(void *ctx, size_t size)
{
PyMemBlockAllocator *ctx;
if (Py_PXCTX)
return _PxMem_Malloc(size))
else
return alloc->malloc(alloc->ctx, size);
}
...
PyMemBlockAllocator pyparallel_pyobject;
static void *
setup_pyparallel_allocator(void)
{
PyMemBlockAllocator alloc;
PyObject_GetAllocator(&pyparallel_pyobject);
alloc.ctx = &pyparallel_pyobject;
alloc.malloc = _PxMem_AllocMalloc;
...
PyObject_SetAllocator(&alloc);
}
-- 8< -
But I don't know if you want pyparallel to be an "optional" feature
chosen at runtime...
Victor
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On Wed, 19 Jun 2013 17:49:02 +0200 Victor Stinner wrote: > 2013/6/19 Antoine Pitrou : > > On Wed, 19 Jun 2013 17:24:21 +0200 > >> >> Drawback: the caller has to check if the result is 0, or handle the > >> >> error. > >> > > >> > Or you can just call Py_FatalError() if the domain is invalid. > >> > >> I don't like Py_FatalError(), especially when Python is embedded. It's > >> safer to return -1 and expect the caller to check for the error case. > > > > I don't think you need to check for errors. The domain is always one of > > the existing constants, i.e. it should be hard-coded in the source, not > > computed. > > Imagine that PyMem_GetBlockAllocator() is part of the stable ABI and > that a new domain is added to Python 3.5. An application is written > for Python 3.5 and is run with Python 3.4: how would the application > notice that PyMem_GetBlockAllocator() does not know the new domain? That's a good question. I don't know why guidelines Martin used when designing the stable ABI, but I would expect important high-level functions to end there, not memory allocation debugging. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pyparallel and new memory API discussions...
On Wed, Jun 19, 2013 at 09:20:15AM -0700, Victor Stinner wrote:
> """
> So, Victor, I'm interested to hear how the new API you're proposing
> will affect this solution I've come up with for pyparallel; I'm
> going to be absolutely dependent upon the ability to lock main
> thread pages as read-only in one fell-swoop -- am I still going to
> be able to do that with your new API in place?
> """
>
> 2013/6/19 Trent Nelson :
> > On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote:
> >> > 1. All memory allocated in a parallel context is localized to a
> >> > private heap.
> >>
> >> How do you allocate memory in this "private" heap? Did you add new
> >> functions to allocate memory?
> >
> > Yup:
> > _PyHeap_Malloc():
> > http://hg.python.org/sandbox/trent/file/0e70a0caa1c0/Python/pyparallel.c#l2365.
> >
> > All memory operations (PyObject_New/Malloc etc) get intercepted
> > during parallel thread execution and redirected to _PyHeap_Malloc(),
> > which is a very simple slab allocator. (No need for convoluted
> > buckets because we never free individual objects during parallel
> > execution; instead, we just blow everything away at the end.)
>
> Ok, so I don't think that the PEP 445 would change anything for you.
>
> The following change might have an impact: If _PyHeap_Malloc is not
> thread safe, replacing PyMem_Malloc() with PyMem_RawMalloc() when the
> GIL is not held would avoid bugs in your code.
Hmmm, well, _PyHeap_Malloc is sort of implicitly thread-safe, by
design, but I'm not sure if we're referring to the same sort of
thread-safe problem here.
For one, _PyHeap_Malloc won't ever run if the GIL isn't being held.
(Parallel threads are only allowed to run when the main thread has
the GIL held and has relinquished control to parallel threads.)
Also, I interpret PyMem_RawMalloc() as a direct shortcut to
malloc() (or something else that returns void *s that are then
free()'d down the track). Is that right?
I don't think that would impact pyparallel.
> If you want to choose dynamically the allocator at runtime, you can
> replace PyObject_Malloc allocator using:
> -- 8< -
> static void *
> _PxMem_AllocMalloc(void *ctx, size_t size)
> {
> PyMemBlockAllocator *ctx;
> if (Py_PXCTX)
> return _PxMem_Malloc(size))
> else
> return alloc->malloc(alloc->ctx, size);
> }
>
> ...
>
> PyMemBlockAllocator pyparallel_pyobject;
>
> static void *
> setup_pyparallel_allocator(void)
> {
> PyMemBlockAllocator alloc;
> PyObject_GetAllocator(&pyparallel_pyobject);
> alloc.ctx = &pyparallel_pyobject;
> alloc.malloc = _PxMem_AllocMalloc;
> ...
> PyObject_SetAllocator(&alloc);
> }
> -- 8< -
>
> But I don't know if you want pyparallel to be an "optional" feature
> chosen at runtime...
Hmmm, those code snippets are interesting. Time for some more
homework.
Trent.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
PyMem_RawAlloc()/Realloc/Free should be part of the stable ABI. I agree that all other new fumctions ans structures should not. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On 20 Jun 2013 02:03, "Victor Stinner" wrote: > > 2013/6/19 Kristján Valur Jónsson : > > Oh, it should be public, in my opinion. > > Ok. And do you think that the PyMemMappingAllocator structure is > complete, or that we should add something to be future-proof? At > least, PyMemMappingAllocator is enough for pymalloc usage :-) > > Is PyMemMappingAllocator complete enough for your usage at CCP Games? Can we go back to calling this the "Arena" allocator? Or at least "Mapped"? When I see "Mapping" in the context of Python I think of the container API, not a memory allocation API. > > > We do exactly that when we embed python into UnrealEngine. We keep pythons internal PyObject_Mem allocator, but have it ask UnrealEngine for its arenas. That way, we can still keep track of python's memory usage from with the larger application, even if the granularity of memory is now on an "arena" level, rather than individual allocs. > > I hope that the PEP 445 is flexible enough to allow you to decide > which functions are hooked and replaced, and which functions will be > leaved unchanged. That's why I'm not in favor of the "Make > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. It's also why I'm in favour of the "domain" API rather than separate functions. 1. In the initial iteration, just have the three basic domains (raw, interpreter, objects). Replacing allocators for third party libraries is the responsibility of embedding applications. 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs so that extension modules can register new domains for wrapped libraries. Replacing allocators is still the responsibility of embedding applications, but there's a consistent API to do it. (Alternatively, we could do both now) And agreed PyMem_Raw* are the only new APIs that should be added to the stable ABI. Cheers, Nick. > > Victor > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
Le jeudi 20 juin 2013, Nick Coghlan a écrit : > > > Is PyMemMappingAllocator complete enough for your usage at CCP Games? > > Can we go back to calling this the "Arena" allocator? Or at least > "Mapped"? When I see "Mapping" in the context of Python I think of the > container API, not a memory allocation API. > This function is written to be able to use mmap() and VirtualAlloc(). There is no Python function to use directly this allocator yet, but I chose "memory mapping" name because it is very different than the heap and it may be useful for other functions than pymalloc. If I change the name, it would be called PyObject_SetArenaAllocator() with a PyObjectArenaAllocator structure. I'm not sure that PyMemMappingAllocator API is future-proof, so I'm fine to call it "arena" again. > > I hope that the PEP 445 is flexible enough to allow you to decide > > which functions are hooked and replaced, and which functions will be > > leaved unchanged. That's why I'm not in favor of the "Make > > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative. > > It's also why I'm in favour of the "domain" API rather than separate > functions. > > 1. In the initial iteration, just have the three basic domains (raw, > interpreter, objects). Replacing allocators for third party libraries is > the responsibility of embedding applications. > > 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs > so that extension modules can register new domains for wrapped libraries. > Replacing allocators is still the responsibility of embedding applications, > but there's a consistent API to do it. > > (Alternatively, we could do both now) > How would you use an allocator of a new domain? PyMemBlockAllocator structure is not convinient, and if Py_GetAllocator() only once, you may loose a hook installed later. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
