Re: [Numpy-discussion] constructing record dtypes from the c-api
Hi Jason, As I understand numpy has been set up to mirror C-structs as long as you use the 'align' flag. For example, your struct can be represented as np.dtype('f8,f4,i4,u8', align=True) (assuming 32 bit floats). The offsets of the fields should be exactly the offsets of the elements of the struct. In C you can create the dtype using PyArray_DescrAlignConverter. If you want to get the field offsets given a numpy dtype in C, you need to iterate through the 'names' and 'fields' attributes of the dtype, eg PyObject *key, *tup, *offobj; n = PyTuple_Size(descr-names); for (i = 0; i n; i++) { key = PyTuple_GetItem(descr-names, i); tup = PyDict_GetItem(descr-fields, key); offobj = PyTuple_GetItem(tup, 1); offset = PyInt_AsSsize_t(offobj); // do something with offset } (error checks DECREFS might be needed) Allan On 07/25/2015 12:26 AM, Jason Newton wrote: After drilling through the sources a second time, I found it was numpy/core/src/multiarray/descriptor.c was the file to consult with the primary routine being PyArray_DescrConverter and _convert_from_* functions being the most interesting to read and glean the capabilities of this with. So in particular it looks like the _convert_from_dict based path is the way to go to allow custom field offsets to safely and transparently map C POD structs with the offset information generated at compile time to hopefully keep dtype's in perfect sync with C sources vs declaring on the python source side. I plan on building a helper class to generate the dictionaries for this subroutine since something akin to the list dtype specification is more user-friendly (even towards me). -Jason On Thu, Jul 23, 2015 at 7:55 PM, Jason Newton nev...@gmail.com wrote: Hi folks, The moderator for the ML approved my subscription so I can now post this back in the numpy list rather than scipy. Apologies for the duplicate/cross posting. I was trying to figure out how to make a dtype for a c-struct on the c-side and storing that in some boost python libraries I'm making. Imagine the following c-struct, greatly simplified of course from the real ones I need to expose: struct datum{ double position[3]; float velocity[3]; int32_t d0; uint64_t time_of_receipt; }; How would you make the dtype/PyArray_Descr for this? I have as a point of reference compound types in HDF for similar struct descriptions (h5py makes these nearly 1:1 and converts back and forth to dtypes and hdf5 types, it uses Cython to accomplish this) - but I don't want to bring in hdf for this task - I'm not sure how well the offsets would go over in that translation to h5py too. Proper/best solutions would make use of offsetof as we insert entries to the dtype/PyArray_Descr. It's fine if you do this in straight C - I just need to figure out how to accomplish this in a practical way. The language I'm working in is C++11. The end goal is probably going to be to create a helper infrastructure to allow this to be made automatically-ish provided implementation of a [static] visitor pattern in C++. The idea is to make numpy compatible c++ POD types rather than use Boost.Python wrapped proxies for every object which will cut down on some complicated and time consuming code (both of computer and developer) when ndarrays are what's called for. Related - would one work with record dtypes passed to C? How would one lookup fields and offsets within a structure? Thanks for any advisement! -Jason ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] constructing record dtypes from the c-api
After drilling through the sources a second time, I found it was numpy/core/src/multiarray/descriptor.c was the file to consult with the primary routine being PyArray_DescrConverter and _convert_from_* functions being the most interesting to read and glean the capabilities of this with. So in particular it looks like the _convert_from_dict based path is the way to go to allow custom field offsets to safely and transparently map C POD structs with the offset information generated at compile time to hopefully keep dtype's in perfect sync with C sources vs declaring on the python source side. I plan on building a helper class to generate the dictionaries for this subroutine since something akin to the list dtype specification is more user-friendly (even towards me). -Jason On Thu, Jul 23, 2015 at 7:55 PM, Jason Newton nev...@gmail.com wrote: Hi folks, The moderator for the ML approved my subscription so I can now post this back in the numpy list rather than scipy. Apologies for the duplicate/cross posting. I was trying to figure out how to make a dtype for a c-struct on the c-side and storing that in some boost python libraries I'm making. Imagine the following c-struct, greatly simplified of course from the real ones I need to expose: struct datum{ double position[3]; float velocity[3]; int32_t d0; uint64_t time_of_receipt; }; How would you make the dtype/PyArray_Descr for this? I have as a point of reference compound types in HDF for similar struct descriptions (h5py makes these nearly 1:1 and converts back and forth to dtypes and hdf5 types, it uses Cython to accomplish this) - but I don't want to bring in hdf for this task - I'm not sure how well the offsets would go over in that translation to h5py too. Proper/best solutions would make use of offsetof as we insert entries to the dtype/PyArray_Descr. It's fine if you do this in straight C - I just need to figure out how to accomplish this in a practical way. The language I'm working in is C++11. The end goal is probably going to be to create a helper infrastructure to allow this to be made automatically-ish provided implementation of a [static] visitor pattern in C++. The idea is to make numpy compatible c++ POD types rather than use Boost.Python wrapped proxies for every object which will cut down on some complicated and time consuming code (both of computer and developer) when ndarrays are what's called for. Related - would one work with record dtypes passed to C? How would one lookup fields and offsets within a structure? Thanks for any advisement! -Jason ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] constructing record dtypes from the c-api
Hi folks, The moderator for the ML approved my subscription so I can now post this back in the numpy list rather than scipy. Apologies for the duplicate/cross posting. I was trying to figure out how to make a dtype for a c-struct on the c-side and storing that in some boost python libraries I'm making. Imagine the following c-struct, greatly simplified of course from the real ones I need to expose: struct datum{ double position[3]; float velocity[3]; int32_t d0; uint64_t time_of_receipt; }; How would you make the dtype/PyArray_Descr for this? I have as a point of reference compound types in HDF for similar struct descriptions (h5py makes these nearly 1:1 and converts back and forth to dtypes and hdf5 types, it uses Cython to accomplish this) - but I don't want to bring in hdf for this task - I'm not sure how well the offsets would go over in that translation to h5py too. Proper/best solutions would make use of offsetof as we insert entries to the dtype/PyArray_Descr. It's fine if you do this in straight C - I just need to figure out how to accomplish this in a practical way. The language I'm working in is C++11. The end goal is probably going to be to create a helper infrastructure to allow this to be made automatically-ish provided implementation of a [static] visitor pattern in C++. The idea is to make numpy compatible c++ POD types rather than use Boost.Python wrapped proxies for every object which will cut down on some complicated and time consuming code (both of computer and developer) when ndarrays are what's called for. Related - would one work with record dtypes passed to C? How would one lookup fields and offsets within a structure? Thanks for any advisement! -Jason ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion