Re: How to specific multiple dtypes in numpy.ndarray?
On Thursday, December 19, 2019 at 2:53:18 AM UTC-8, lampahome wrote:
> I meet performance is low when I use struct.unpack to unpack binary data.
>
> So I tried to use numpy.ndarray
> But meet error when I want to unpack multiple dtypes
>
> Can anyone teach me~
>
> Code like below:
> # python3
> import struct
> import numpy as np
> s1 = struct.Struct("@QIQ")
> ss1 = s1.pack(1,11,111)
> np.ndarray((3,), [('Q','I','Q')], ss1)
> # ValueError: mismatch in size of old and new data-descriptor.
Is there some reason that you're calling the np.ndarray constructor directly
instead of the np.array convenience function? The arguments that you pass to
each are slightly different. Generally, you want to use np.array.
The first argument to np.ndarray is the shape. You appear to want three
elements.
The second argument to np.ndarray is the data type (dtype). It needs to be a
SINGLE data type, like 'float' or 'int' or 'object'. All elements in in array
are of the same dtype. If you want different behavior, I think that you want
lists, not arrays. You can set the dtype to 'object' and you should be able to
store any data type as an array element. But I don't think that offers you any
advantages over using ordinary Python lists.
In any case, it appears that you aren't passing a dtype as your second
argument. You're trying to pass the data itself, as a tuple wrapped in a
single list. This seems to be certain to raise an error.
The third argument to np.ndarray is supposed to be the buffer which contains
your data.
https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html#numpy.array
ile:///usr/share/doc/python-numpy-doc/html/reference/generated/numpy.array.html#numpy.array
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to specific multiple dtypes in numpy.ndarray?
On 19/12/2019 11.52, lampahome wrote:
> I meet performance is low when I use struct.unpack to unpack binary data.
>
> So I tried to use numpy.ndarray
> But meet error when I want to unpack multiple dtypes
>
> Can anyone teach me~
>
> Code like below:
> # python3
> import struct
> import numpy as np
> s1 = struct.Struct("@QIQ")
> ss1 = s1.pack(1,11,111)
> np.ndarray((3,), [('Q','I','Q')], ss1)
> # ValueError: mismatch in size of old and new data-descriptor.
A numpy array always has ONE dtype for ALL array elements.
If you read an array of structs, you can define a structured type, where
each element of your struct must have a name.
The error you're seeing is (as you know) because you're not setting up
your dtype in the right way. Let's fix it:
> In [2]: np.dtype([('Q', 'I',
> 'Q')])
>
>
> ---
> ValueError Traceback (most recent call
> last)
> in
> > 1 np.dtype([('Q', 'I', 'Q')])
>
> ValueError: mismatch in size of old and new data-descriptor
>
> In [3]: np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3',
> 'Q')])
> Out[3]: dtype([('field1', '
> In [4]:
>
>
... and now let's put it all together!
s1 = struct.Struct("@QIQ")
ss1 = s1.pack(1,11,111)
struct_dtype = np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3', 'Q')])
a = np.frombuffer(ss1, dtype=struct_dtype)
I'm using the frombuffer() function deliberately so I don't have to
figure out the shape of the final array (which is (1,), not (3,), by the
way).
And hey presto: it raises an exception!
> ValueError: buffer size must be a multiple of element size
Your example shows a difference between the default behaviour of numpy's
structured dtype and the struct module: packing! By default, numpy
structured dtypes are closely packed, i.e. nothing is aligned to useful
memory boundaries.
struct_type.itemsize == 20
The struct module, on the other hand, tries to guess where the C
compiler would put its padding.
len(ss1) == 24
We can tell numpy to do the same:
struct_dtype = np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3',
'Q')], align=True)
and then
a = np.frombuffer(ss1, dtype=struct_dtype)
works and produces
array([(1, 11, 111)],
dtype={'names':['field1','field2','field3'],
'formats':['https://mail.python.org/mailman/listinfo/python-list
How to specific multiple dtypes in numpy.ndarray?
I meet performance is low when I use struct.unpack to unpack binary data.
So I tried to use numpy.ndarray
But meet error when I want to unpack multiple dtypes
Can anyone teach me~
Code like below:
# python3
import struct
import numpy as np
s1 = struct.Struct("@QIQ")
ss1 = s1.pack(1,11,111)
np.ndarray((3,), [('Q','I','Q')], ss1)
# ValueError: mismatch in size of old and new data-descriptor.
--
https://mail.python.org/mailman/listinfo/python-list
