On Mon, Dec 14, 2020 at 4:46 PM Kevin Sheppard <kevin.k.shepp...@gmail.com> wrote: > > You need to reacquire the gil, then you can get the lock and rerelease the > gil. > > I think this works (on phone, so untested) > > with gil: > with nogil, lock: > ...
Thanks Kevin. This surely works, but feels seriously weird. Is this the only / the recommended way? I can of course adjust the buffer size to amortize the time for the GIL manipulation, but this really starts looking like a code smell. My original motivation was to have inner simulation loops python-free. Most likely, the issue is that I'm not using the Generator correctly? Evgeni > On Mon, Dec 14, 2020, 13:37 Evgeni Burovski <evgeny.burovs...@gmail.com> > wrote: >> >> Hi, >> >> What would be the correct way of locking the bit generator of >> np.random.Generator in cython's nogil context? >> (This is threading 101, surely, so please forgive my ignorance). >> >> The docs for extending np.random.Generator in cython >> (https://numpy.org/doc/stable/reference/random/extending.html#cython) >> recommend the following idiom for generating uniform variates, where >> the GIL is released and a Generator-specific lock is held: >> >> x = PCG64() >> rng = <bitgen_t *> PyCapsule_GetPointer(x.capsule, capsule_name) >> with nogil, x.lock: >> rng.next_double(rng.state) >> >> What is the correct way of locking it when already in the nogil >> section (so that x.lock is not accessible)? >> >> The use case is a long-running MC process which generates random >> variates in a tight loop (so the loop is all nogil). In case it >> matters, I probably won't be using python threads, but may use >> multiprocessing. >> >> Basically, >> >> cdef double uniform(self) nogil: >> if self.idx >= self.buf.shape[0]: >> self._fill() >> cdef double value = self.buf[self.idx] >> self.idx += 1 >> return value >> >> cdef void _fill(self) nogil: >> self.idx = 0 >> # HERE: Lock ? >> for i in range(self.buf.shape[0]): >> self.buf[i] = self.rng.next_double(self.rng.state) >> >> >> Thanks, >> Evgeni >> >> >> P.S. The full cdef class, for completeness: >> >> cdef class RndmWrapper(): >> cdef: >> double[::1] buf >> Py_ssize_t idx >> bitgen_t *rng >> object py_gen # keep the garbage collector away >> >> def __init__(self, seed=(1234, 0), buf_size=4096, bitgen_kind=None): >> if bitgen_kind is None: >> bitgen_kind = PCG64 >> >> # cf Numpy-discussion list, K.~Sheppard, R.~Kern, June 29, >> 2020 and below >> # >> https://mail.python.org/pipermail/numpy-discussion/2020-June/080794.html >> entropy, num = seed >> seed_seq = SeedSequence(entropy, spawn_key=(num,)) >> py_gen = bitgen_kind(seed_seq) >> >> # store the python object to avoid it being garbage collected >> self.py_gen = py_gen >> >> capsule = py_gen.capsule >> self.rng = <bitgen_t *>PyCapsule_GetPointer(capsule, capsule_name) >> if not PyCapsule_IsValid(capsule, capsule_name): >> raise ValueError("Invalid pointer to anon_func_state") >> >> self.buf = np.empty(buf_size, dtype='float64') >> self._fill() >> >> @cython.boundscheck(False) >> @cython.wraparound(False) >> cdef void _fill(self) nogil: >> self.idx = 0 >> for i in range(self.buf.shape[0]): >> self.buf[i] = self.rng.next_double(self.rng.state) >> >> @cython.boundscheck(False) >> @cython.wraparound(False) >> cdef double uniform(self) nogil: >> if self.idx >= self.buf.shape[0]: >> self._fill() >> cdef double value = self.buf[self.idx] >> self.idx += 1 >> return value >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
