On 03.06.2013 14:43, Andreas Hilboll wrote:
> Hi,
> 
> I'm storing large datasets (5760 x 2880 x ~150) in a compressed EArray
> (the last dimension represents time, and once per month there'll be one
> more 5760x2880 array to add to the end).
> 
> Now, extracting timeseries at one index location is slow; e.g., for four
> indices, it takes several seconds:
> 
>    In [19]: idx = ((5000, 600, 800, 900), (1000, 2000, 500, 1))
> 
>    In [20]: %time AA = np.vstack([_a[i,j] for i,j in zip(*idx)])
>    CPU times: user 4.31 s, sys: 0.07 s, total: 4.38 s
>    Wall time: 7.17 s
> 
> I have the feeling that this performance could be improved, but I'm not
> sure about how to properly use the `chunkshape` parameter in my case.
> 
> Any help is greatly appreciated :)
> 
> Cheers, Andreas.

PS: If I could get significant performance gains by not using an EArray
and therefore re-creating the whole database each month, then this would
also be an option.

-- Andreas.


------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite
It's a free troubleshooting tool designed for production
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Reply via email to