I was interested in how long it takes to iterate, since this is arguably
where the
majority of the time is spent.
On Thu, Apr 19, 2012 at 8:43 AM, Alvaro Tejero Cantero <alv...@minin.es>wrote:
> Some complementary info (I copy the details of the tables below)
>
> timeit vals = numpy.fromiter((x['val'] for x in
> my.root.raw.t0.wtab02.where('val>1')),dtype=np.int16)
> 1 loops, best of 3: 30.4 s per loop
>
>
> Using the compressed and indexed version, it mysteriously does not
> work (output is empty list)
> >>> cvals = np.fromiter((x['val'] for x in wctab02.where('val>1')),
> dtype=np.int16)
> >>> cvals
> array([], dtype=int16)
>
This doesn't work because numpy doesn't accept generators. The following
should work:
>>> cvals = np.fromiter([x['val'] for x in wctab02.where('val>1')],
dtype=np.int16)
Also, I am a little concerned that np.nonzero() doesn't really compare to
Table.getWhereList('val>1'). Testing for all zero bits *should be* a lot
faster
than a numeric comparison. Could you instead try the same actual operation
in numpy as whereList():
>>> timeit big=np.argwhere(np.greater(wa02[:], 1))
Thanks!
Anthony
>
> But it does if we skip using where ( I don't print cvals, but it is
> correct )
> >>> timeit cvals = np.fromiter((x['val'] for x in wctab02 if x['val']>1),
> dtype=np.int16)
> 1 loops, best of 3: 54.8 s per loop
>
> (the version with longer chunklen works fine and times to 30.7s).
>
>
> -á.
>
> wtab02: not compressed, not indexed, small chunklen:
> /raw/t0/wtab02 (Table(312000000,)) ''
> description := {
> "val": Int16Col(shape=(), dflt=0, pos=0)}
> byteorder := 'little'
> chunkshape := (32768,)
>
> larger chunklen (as calculated from expectedrows=312000000)
> /raw/t0/wcetab02 (Table(312000000,)) 'test'
> description := {
> "val": Int16Col(shape=(), dflt=0, pos=0)}
> byteorder := 'little'
> chunkshape := (131072,)
>
> wctab02: compressed, with CSI index
> /raw/t0/wctab02 (Table(312000000,), shuffle, blosc(9)) 'test'
> description := {
> "val": Int16Col(shape=(), dflt=0, pos=0)}
> byteorder := 'little'
> chunkshape := (32768,)
> autoIndex := True
> colindexes := {
> "val": Index(9, full, shuffle, zlib(1)).is_CSI=True}
>
>
>
> On Thu, Apr 19, 2012 at 12:46, Alvaro Tejero Cantero <alv...@minin.es>
> wrote:
> > where will give me an iterator over the /values/; in this case I
> > wanted the indexes. Plus, it will give me an iterator, so it will be
> > trivially fast.
> >
> > Are you interested in the timings of where + building a list? or where
> > + building an array?
> >
> >
> > -á.
> >
> >
> >
> > On Wed, Apr 18, 2012 at 19:02, Anthony Scopatz <scop...@gmail.com>
> wrote:
> >>
>
>
> ------------------------------------------------------------------------------
> For Developers, A Lot Can Happen In A Second.
> Boundary is the first to Know...and Tell You.
> Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
> http://p.sf.net/sfu/Boundary-d2dvs2
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
