<snip>
> I also think that the lock only matters for Multithreaded code not
> Multiprocess. I believe the latter pickles and unpickles any Generator
> object (and the underying BitGenerator) and so each process has its own
> version. Note that when multiprocessing the recommended procedure is to use
> spawn() to generate a sequence of BitGenerators and to use a distinct
> BitGenerator in each process. If you do this then you are free from the lock.
Thanks. Just to confirm: does using SeedSequence spawn_key arg
generate distinct BitGenerators? As in
cdef class Wrapper():
def __init__(self, seed):
entropy, num = seed
py_gen = PCG64(SeedSequence(entropy, spawn_key=(spawn_key,)))
self.rng = <bitgen_t *>
py_gen.capsule.PyCapsule_GetPointer(capsule, "BitGenerator") # <---
this
cdef Wrapper rng_0 = Wrapper(seed=(123, 0))
cdef Wrapper rng_1 = Wrapper(seed=(123, 1))
And then,of these two objects, do they have distinct BitGenerators?
Evgeni
>
>
> Kevin
>
>
>
>
>
> From: Evgeni Burovski
> Sent: Monday, December 14, 2020 2:10 PM
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion] locking np.random.Generator in a cython nogil
> context?
>
>
>
> On Mon, Dec 14, 2020 at 4:46 PM Kevin Sheppard
>
> <[email protected]> wrote:
>
> >
>
> > You need to reacquire the gil, then you can get the lock and rerelease the
> > gil.
>
> >
>
> > I think this works (on phone, so untested)
>
> >
>
> > with gil:
>
> > with nogil, lock:
>
> > ..
>
>
>
> Thanks Kevin.
>
> This surely works, but feels seriously weird. Is this the only / the
>
> recommended way?
>
> I can of course adjust the buffer size to amortize the time for the
>
> GIL manipulation, but this really starts looking like a code smell.
>
> My original motivation was to have inner simulation loops python-free.
>
> Most likely, the issue is that I'm not using the Generator correctly?
>
>
>
> Evgeni
>
>
>
>
>
> > On Mon, Dec 14, 2020, 13:37 Evgeni Burovski <[email protected]>
> > wrote:
>
> >>
>
> >> Hi,
>
> >>
>
> >> What would be the correct way of locking the bit generator of
>
> >> np.random.Generator in cython's nogil context?
>
> >> (This is threading 101, surely, so please forgive my ignorance).
>
> >>
>
> >> The docs for extending np.random.Generator in cython
>
> >> (https://numpy.org/doc/stable/reference/random/extending.html#cython)
>
> >> recommend the following idiom for generating uniform variates, where
>
> >> the GIL is released and a Generator-specific lock is held:
>
> >>
>
> >> x = PCG64()
>
> >> rng = <bitgen_t *> PyCapsule_GetPointer(x.capsule, capsule_name)
>
> >> with nogil, x.lock:
>
> >> rng.next_double(rng.state)
>
> >>
>
> >> What is the correct way of locking it when already in the nogil
>
> >> section (so that x.lock is not accessible)?
>
> >>
>
> >> The use case is a long-running MC process which generates random
>
> >> variates in a tight loop (so the loop is all nogil). In case it
>
> >> matters, I probably won't be using python threads, but may use
>
> >> multiprocessing.
>
> >>
>
> >> Basically,
>
> >>
>
> >> cdef double uniform(self) nogil:
>
> >> if self.idx >= self.buf.shape[0]:
>
> >> self._fill()
>
> >> cdef double value = self.buf[self.idx]
>
> >> self.idx += 1
>
> >> return value
>
> >>
>
> >> cdef void _fill(self) nogil:
>
> >> self.idx = 0
>
> >> # HERE: Lock ?
>
> >> for i in range(self.buf.shape[0]):
>
> >> self.buf[i] = self.rng.next_double(self.rng.state)
>
> >>
>
> >>
>
> >> Thanks,
>
> >> Evgeni
>
> >>
>
> >>
>
> >> P.S. The full cdef class, for completeness:
>
> >>
>
> >> cdef class RndmWrapper():
>
> >> cdef:
>
> >> double[::1] buf
>
> >> Py_ssize_t idx
>
> >> bitgen_t *rng
>
> >> object py_gen # keep the garbage collector away
>
> >>
>
> >> def __init__(self, seed=(1234, 0), buf_size=4096, bitgen_kind=None):
>
> >> if bitgen_kind is None:
>
> >> bitgen_kind = PCG64
>
> >>
>
> >> # cf Numpy-discussion list, K.~Sheppard, R.~Kern, June 29,
>
> >> 2020 and below
>
> >> #
> >> https://mail.python.org/pipermail/numpy-discussion/2020-June/080794.html
>
> >> entropy, num = seed
>
> >> seed_seq = SeedSequence(entropy, spawn_key=(num,))
>
> >> py_gen = bitgen_kind(seed_seq)
>
> >>
>
> >> # store the python object to avoid it being garbage collected
>
> >> self.py_gen = py_gen
>
> >>
>
> >> capsule = py_gen.capsule
>
> >> self.rng = <bitgen_t *>PyCapsule_GetPointer(capsule, capsule_name)
>
> >> if not PyCapsule_IsValid(capsule, capsule_name):
>
> >> raise ValueError("Invalid pointer to anon_func_state")
>
> >>
>
> >> self.buf = np.empty(buf_size, dtype='float64')
>
> >> self._fill()
>
> >>
>
> >> @cython.boundscheck(False)
>
> >> @cython.wraparound(False)
>
> >> cdef void _fill(self) nogil:
>
> >> self.idx = 0
>
> >> for i in range(self.buf.shape[0]):
>
> >> self.buf[i] = self.rng.next_double(self.rng.state)
>
> >>
>
> >> @cython.boundscheck(False)
>
> >> @cython.wraparound(False)
>
> >> cdef double uniform(self) nogil:
>
> >> if self.idx >= self.buf.shape[0]:
>
> >> self._fill()
>
> >> cdef double value = self.buf[self.idx]
>
> >> self.idx += 1
>
> >> return value
>
> >> _______________________________________________
>
> >> NumPy-Discussion mailing list
>
> >> [email protected]
>
> >> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> >
>
> > _______________________________________________
>
> > NumPy-Discussion mailing list
>
> > [email protected]
>
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
>
> NumPy-Discussion mailing list
>
> [email protected]
>
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[email protected]
https://mail.python.org/mailman/listinfo/numpy-discussion
