[Pytables-users] Reading multiple tables on multithread

Hi, I've been trying to open an H5 file from multiple processes as read-only and each process reads a different table. If I do this I get errors so I suspect that pytables doesn't support this ? I think that this shouldn't be a problem as I'm opening the file as read-only so it won't corrupt the

[Pytables-users] storing long 1-d arrays

Hi all, I'm thinking of using pytables to efficiently store (and manage) long arrays. In my case each dataset would contain two dozens of arrays each 50 M - 250 M long. for each of them I know the exact length before any operation so I thought I can use CArray (to compress fixed sized arrays). T

Re: [Pytables-users] Reading multiple tables on multithread

A Thursday 03 March 2011 16:33:37 Felipe Barriga Richards escrigué: > Hi, > > I've been trying to open an H5 file from multiple processes as > read-only and each process reads a different table. If I do this I > get errors so I suspect that pytables doesn't support this ? > > I think that this sh

Re: [Pytables-users] comparing pytables with memmory mapped arrays

A Thursday 03 March 2011 17:32:05 samuel sinayoko escrigué: > Hi everyone, > > I'm a postdoc in fluid dynamics and acoustics. I need to compute > Fourier transforms of big arrays (~4GB). I've been comparing various > options to do so: > - option 1: memmory mapped arrays (with numpy), using the fir

Re: [Pytables-users] storing long 1-d arrays

A Thursday 03 March 2011 19:23:25 Davide Cittaro escrigué: > Hi all, I'm thinking of using pytables to efficiently store (and > manage) long arrays. In my case each dataset would contain two > dozens of arrays each 50 M - 250 M long. for each of them I know the > exact length before any operation s

Re: [Pytables-users] storing long 1-d arrays

On 03/mar/2011, at 20:23, Francesc Alted wrote: > You said that your data is 1-d, so why do you insist in specifying 1 2-d > shape? I'd go saving 1-d straight. > Mmm, I've probably missed a 1d example in the docs, I hence thought I should specify 2d shape (1 col). Thanks d --

Re: [Pytables-users] storing long 1-d arrays

On 03/mar/2011, at 20:42, Davide Cittaro wrote: > Mmm, I've probably missed a 1d example in the docs, I hence thought I should > specify 2d shape (1 col). > Thanks Do you mean shape=(N,0) ? d -- Free Software Downl

Re: [Pytables-users] storing long 1-d arrays

A Thursday 03 March 2011 20:46:30 Davide Cittaro escrigué: > On 03/mar/2011, at 20:42, Davide Cittaro wrote: > > Mmm, I've probably missed a 1d example in the docs, I hence thought > > I should specify 2d shape (1 col). Thanks > > Do you mean shape=(N,0) ? Well, if you want to save an array of s

Re: [Pytables-users] storing long 1-d arrays

On 3 mrt 2011, at 20:46, Davide Cittaro wrote: > On 03/mar/2011, at 20:42, Davide Cittaro > wrote: > >> Mmm, I've probably missed a 1d example in the docs, I hence thought I should >> specify 2d shape (1 col). >> Thanks > > Do you mean shape=(N,0) ? shape=(N,) as per standard Python Maar

Re: [Pytables-users] comparing pytables with memmory mapped arrays

Dear Francesc, Thanks for the advise. I had tried some tutorials on PyTables and was under the impression that you could get higher performance than with memmory mapped arrays. I think I'm starting to get a better feel now of the respective advantages of memmory mapped arrays vs pytables. In my ca

Re: [Pytables-users] storing long 1-d arrays

On Mar 3, 2011, at 8:54 PM, Francesc Alted wrote: > > > a = np.arange(1000, dtype=np.float64) > ca = h5f.createCArray(f.root, 'a', tb.Float64Atom(), a.shape) > > Does that make sense? > it does, thanks d /* Davide Cittaro, PhD Cogentech - Consortium for Genomic Technologies via adamello, 1

Re: [Pytables-users] storing long 1-d arrays

On Mar 3, 2011, at 8:55 PM, Maarten Sneep wrote: > > shape=(N,) > > as per standard Python The fact is I never set the shape in numpy for 1d arrays... :-) d /* Davide Cittaro, PhD Cogentech - Consortium for Genomic Technologies via adamello, 16 20139 Milano Italy tel.: +39(02)574303007 e-ma