Re: [Numpy-discussion] mtrand.c update 1.11 breaks my crappy code

[Robert Kern]

On Wed, Apr 6, 2016 at 4:17 PM, Neal Becker  wrote:

> I prefer to use a single instance of a RandomState so that there are
> guarantees about the independence of streams generated from python random
> functions, and from my c++ code.  True, there are simpler approaches - but
> I'm a purist.

Consider using PRNGs that actually expose truly independent streams instead
of a single shared stream:

https://github.com/bashtage/ng-numpy-randomstate

--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] mtrand.c update 1.11 breaks my crappy code

[Neal Becker]

Nathaniel Smith wrote:

> On Apr 6, 2016 06:31, "Robert Kern"  wrote:
>>
>> On Wed, Apr 6, 2016 at 2:18 PM, Neal Becker  wrote:
>> >
>> > I have C++ code that tries to share the mtrand state.  It unfortunately
>> > depends on the layout of RandomState which used to be:
>> >
>> > struct __pyx_obj_6mtrand_RandomState {
>> >   PyObject_HEAD
>> >   rk_state *internal_state;
>> >   PyObject *lock;
>> > };
>> >
>> > But with 1.11 it's:
>> > struct __pyx_obj_6mtrand_RandomState {
>> >   PyObject_HEAD
>> >   struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
>> >   rk_state *internal_state;
>> >   PyObject *lock;
>> >   PyObject *state_address;
>> > };
>> >
>> > So
>> > 1. Why the change?
>> > 2. How can I write portable code?
>>
>> There is no C API to RandomState at this time, stable, portable or
> otherwise. It's all private implementation detail. If you would like a
> stable and portable C API for RandomState, you will need to contribute one
> using PyCapsules to expose the underlying rk_state* pointer.
>>
>> https://docs.python.org/2.7/c-api/capsule.html
> 
> I'm very wary about the idea of exposing the rk_state pointer at all. We
> could have a C API to random but my strong preference would be for
> something that only exposes opaque function calls that take a RandomState
> and return some random numbers, and getting even this right in a clean and
> maintainable way isn't trivial.
> 
> Obviously another option is to call one of the python methods to get an
> ndarray and read out its memory contents. If you can do this in a batch
> (fetching a bunch of numbers for each call) to amortize the additional
> overhead of going through python, then it might work fine. (Python
> overhead is not actually that much -- mostly just having to do a handful
> of extra allocations.)
> 
> Or, possibly the best option, one could use one of the many fine C random
> libraries inside your code, and if you need your code to be deterministic
> given a RandomState you could derive your state initialization from a
> single call to some RandomState method.
> 
> -n

I prefer to use a single instance of a RandomState so that there are 
guarantees about the independence of streams generated from python random 
functions, and from my c++ code.  True, there are simpler approaches - but 
I'm a purist.

Yes, if there were an api use mkl random functions from a RandomState object 
that would solve my problem.  Or even if there was an API to get a 
internal_state pointer from a RandomState object.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] mtrand.c update 1.11 breaks my crappy code

[Nathaniel Smith]

On Apr 6, 2016 06:31, "Robert Kern"  wrote:
>
> On Wed, Apr 6, 2016 at 2:18 PM, Neal Becker  wrote:
> >
> > I have C++ code that tries to share the mtrand state.  It unfortunately
> > depends on the layout of RandomState which used to be:
> >
> > struct __pyx_obj_6mtrand_RandomState {
> >   PyObject_HEAD
> >   rk_state *internal_state;
> >   PyObject *lock;
> > };
> >
> > But with 1.11 it's:
> > struct __pyx_obj_6mtrand_RandomState {
> >   PyObject_HEAD
> >   struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
> >   rk_state *internal_state;
> >   PyObject *lock;
> >   PyObject *state_address;
> > };
> >
> > So
> > 1. Why the change?
> > 2. How can I write portable code?
>
> There is no C API to RandomState at this time, stable, portable or
otherwise. It's all private implementation detail. If you would like a
stable and portable C API for RandomState, you will need to contribute one
using PyCapsules to expose the underlying rk_state* pointer.
>
> https://docs.python.org/2.7/c-api/capsule.html

I'm very wary about the idea of exposing the rk_state pointer at all. We
could have a C API to random but my strong preference would be for
something that only exposes opaque function calls that take a RandomState
and return some random numbers, and getting even this right in a clean and
maintainable way isn't trivial.

Obviously another option is to call one of the python methods to get an
ndarray and read out its memory contents. If you can do this in a batch
(fetching a bunch of numbers for each call) to amortize the additional
overhead of going through python, then it might work fine. (Python overhead
is not actually that much -- mostly just having to do a handful of extra
allocations.)

Or, possibly the best option, one could use one of the many fine C random
libraries inside your code, and if you need your code to be deterministic
given a RandomState you could derive your state initialization from a
single call to some RandomState method.

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] mtrand.c update 1.11 breaks my crappy code

[Neal Becker]

Neal Becker wrote:

> Robert Kern wrote:
> 
>> On Wed, Apr 6, 2016 at 2:18 PM, Neal Becker  wrote:
>>>
>>> I have C++ code that tries to share the mtrand state.  It unfortunately
>>> depends on the layout of RandomState which used to be:
>>>
>>> struct __pyx_obj_6mtrand_RandomState {
>>>   PyObject_HEAD
>>>   rk_state *internal_state;
>>>   PyObject *lock;
>>> };
>>>
>>> But with 1.11 it's:
>>> struct __pyx_obj_6mtrand_RandomState {
>>>   PyObject_HEAD
>>>   struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
>>>   rk_state *internal_state;
>>>   PyObject *lock;
>>>   PyObject *state_address;
>>> };
>>>
>>> So
>>> 1. Why the change?
>>> 2. How can I write portable code?
>> 
>> There is no C API to RandomState at this time, stable, portable or
>> otherwise. It's all private implementation detail. If you would like a
>> stable and portable C API for RandomState, you will need to contribute
>> one using PyCapsules to expose the underlying rk_state* pointer.
>> 
>> https://docs.python.org/2.7/c-api/capsule.html
>> 
>> --
>> Robert Kern
> 
> I don't see how pycapsule helps here.  What I need is, my C++ code
> receives
> a RandomState object.  I need to call e.g., rk_random, passing the pointer
> to rk_state - code looks like this;
> 
> RandomState* r = (RandomState*)(rs.ptr());
> //result_type buffer;
> //rk_fill ((void*), sizeof(buffer), r->internal_state);
> if (sizeof(result_type) == sizeof (uint64_t))
>   return rk_ulong (r->internal_state);
> else if (sizeof(result_type) == sizeof (uint32_t))
>   return rk_random (r->internal_state);

Nevermind, I see it's described here:
https://docs.python.org/2.7/extending/extending.html#using-capsules


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] mtrand.c update 1.11 breaks my crappy code

[Neal Becker]

Robert Kern wrote:

> On Wed, Apr 6, 2016 at 2:18 PM, Neal Becker  wrote:
>>
>> I have C++ code that tries to share the mtrand state.  It unfortunately
>> depends on the layout of RandomState which used to be:
>>
>> struct __pyx_obj_6mtrand_RandomState {
>>   PyObject_HEAD
>>   rk_state *internal_state;
>>   PyObject *lock;
>> };
>>
>> But with 1.11 it's:
>> struct __pyx_obj_6mtrand_RandomState {
>>   PyObject_HEAD
>>   struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
>>   rk_state *internal_state;
>>   PyObject *lock;
>>   PyObject *state_address;
>> };
>>
>> So
>> 1. Why the change?
>> 2. How can I write portable code?
> 
> There is no C API to RandomState at this time, stable, portable or
> otherwise. It's all private implementation detail. If you would like a
> stable and portable C API for RandomState, you will need to contribute one
> using PyCapsules to expose the underlying rk_state* pointer.
> 
> https://docs.python.org/2.7/c-api/capsule.html
> 
> --
> Robert Kern

I don't see how pycapsule helps here.  What I need is, my C++ code receives 
a RandomState object.  I need to call e.g., rk_random, passing the pointer 
to rk_state - code looks like this;

RandomState* r = (RandomState*)(rs.ptr());
//result_type buffer;
//rk_fill ((void*), sizeof(buffer), r->internal_state);
if (sizeof(result_type) == sizeof (uint64_t))
  return rk_ulong (r->internal_state);
else if (sizeof(result_type) == sizeof (uint32_t))
  return rk_random (r->internal_state);

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] mtrand.c update 1.11 breaks my crappy code

[Robert Kern]

On Wed, Apr 6, 2016 at 2:18 PM, Neal Becker  wrote:
>
> I have C++ code that tries to share the mtrand state.  It unfortunately
> depends on the layout of RandomState which used to be:
>
> struct __pyx_obj_6mtrand_RandomState {
>   PyObject_HEAD
>   rk_state *internal_state;
>   PyObject *lock;
> };
>
> But with 1.11 it's:
> struct __pyx_obj_6mtrand_RandomState {
>   PyObject_HEAD
>   struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
>   rk_state *internal_state;
>   PyObject *lock;
>   PyObject *state_address;
> };
>
> So
> 1. Why the change?
> 2. How can I write portable code?

There is no C API to RandomState at this time, stable, portable or
otherwise. It's all private implementation detail. If you would like a
stable and portable C API for RandomState, you will need to contribute one
using PyCapsules to expose the underlying rk_state* pointer.

https://docs.python.org/2.7/c-api/capsule.html

--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion