Some complementary info (I copy the details of the tables below)
timeit vals = numpy.fromiter((x['val'] for x in
my.root.raw.t0.wtab02.where('val>1')),dtype=np.int16)
1 loops, best of 3: 30.4 s per loop
Using the compressed and indexed version, it mysteriously does not
work (output is empty list)
>>> cvals = np.fromiter((x['val'] for x in wctab02.where('val>1')),
>>> dtype=np.int16)
>>> cvals
array([], dtype=int16)
But it does if we skip using where ( I don't print cvals, but it is correct )
>>> timeit cvals = np.fromiter((x['val'] for x in wctab02 if x['val']>1),
>>> dtype=np.int16)
1 loops, best of 3: 54.8 s per loop
(the version with longer chunklen works fine and times to 30.7s).
-á.
wtab02: not compressed, not indexed, small chunklen:
/raw/t0/wtab02 (Table(312000000,)) ''
description := {
"val": Int16Col(shape=(), dflt=0, pos=0)}
byteorder := 'little'
chunkshape := (32768,)
larger chunklen (as calculated from expectedrows=312000000)
/raw/t0/wcetab02 (Table(312000000,)) 'test'
description := {
"val": Int16Col(shape=(), dflt=0, pos=0)}
byteorder := 'little'
chunkshape := (131072,)
wctab02: compressed, with CSI index
/raw/t0/wctab02 (Table(312000000,), shuffle, blosc(9)) 'test'
description := {
"val": Int16Col(shape=(), dflt=0, pos=0)}
byteorder := 'little'
chunkshape := (32768,)
autoIndex := True
colindexes := {
"val": Index(9, full, shuffle, zlib(1)).is_CSI=True}
On Thu, Apr 19, 2012 at 12:46, Alvaro Tejero Cantero <alv...@minin.es> wrote:
> where will give me an iterator over the /values/; in this case I
> wanted the indexes. Plus, it will give me an iterator, so it will be
> trivially fast.
>
> Are you interested in the timings of where + building a list? or where
> + building an array?
>
>
> -á.
>
>
>
> On Wed, Apr 18, 2012 at 19:02, Anthony Scopatz <scop...@gmail.com> wrote:
>>
------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
