Re: [Pytables-users] Numpy Arrays to Structure Array or Table

[Anthony Scopatz]

Hi David,

I think that you can do what you want in one, rather long line:

hfile.createTable(grp, 'signal', description=np.array(zip(some_func(t, v)),
dtype=[('time', np.float64), ('value', np.float64)]))

Or two nicer lines:

arr = np.array(zip(some_func(t, v)), dtype=[('time', np.float64), ('value',
np.float64)])
hfile.createTable(grp, 'signal', description=arr)

zip() is your friend =).  If zip is too slow and you don't want to make
more than one copy, you could try something like this:

temparr = np.array(some_func(t, v)).T
arr = np.view(temparr, dtype=[('time', np.float64), ('value', np.float64)])

This really only works because both columns have the same dtype.

Of course, you can always keep basically what you have and loop through the
column names programmaticly:

for name, col in zip(A.dtype.names, some_func(t, v)):
A[name] = col

I hope this helps!

Be Well
Anthony

On Wed, Aug 7, 2013 at 5:58 PM, David Reed david.ree...@gmail.com wrote:

 Hi there,

 I have some generic functions that take time series data with 2 numpy
 array arguments, time and value, and return 2 numpy arrays of time and
 value.

 I would like to place these arrays into a Numpy structured array or
 directly into a new pytables table with fields, time and value.

 Now Ive found I could do this:

 t, v = some_func(t, v)

 A = np.empty(len(t), dtype=[('time', np.float64), ('value',
 np.float64)])

 A['time'] = t
 A['value'] = v

 hfile.createTable(grp, 'signal', description=A)
 hfile.flush()

 But this seems rather clunky and inefficient.  Any suggestions to make
 this repackaging a little smoother?




 --
 Get 100% visibility into Java/.NET code with AppDynamics Lite!
 It's a free troubleshooting tool designed for production.
 Get down to code-level detail for bottlenecks, with 2% overhead.
 Download for free and get started troubleshooting in minutes.
 http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk
 ___
 Pytables-users mailing list
 Pytables-users@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/pytables-users


--
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with 2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] searching for group names

[Gabriel J.L. Beckers]

Anthony Scopatz scop...@gmail.com schreef:

 Are you using compression on this EArray?  This method is basically a thin
 wrapper over some HDF5 functions. I think that the data that you are asking
 for (inadvertently, maybe) is just expensive to get.

No, no compression. But I saw this is one of the first pytables data  
sets I created years ago. The chunk size was not chosen well. I  
improved that now (better chunk size/shape, transposed axes, and using  
CArray) and things are roughly 50% faster.

But I still don't understand why so much data is apparently being read  
when I only want to know which children (i.e. the leaf names) a group  
contains. To do this in my program I loop over _v_children.items(),  
i.e., like,

d = {}
for label, node in f.root.recordings.AB_5000._v_children.items():
d[label] = node

I would have expected code like this to yield a dictionary with node  
objects, without reading/inspecting the data content that nodes  
contain. But apparently under the hood HDF5 is looking at the contents  
of the nodes, which takes a while if they are large, especially over a  
usb3 connection. It is not reading the full array into RAM, because  
the memory footprint of the python session doesn't increase  
appreciably if I run the code above.

Thanks, all the best, Gabriel


--
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with 2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031iu=/4140/ostg.clktrk
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users