Hi all,
I have a table that looks like this:
/table (Table(17801755,), shuffle, zlib(1)) ''
description := {
"field1": UInt32Col(shape=(), dflt=0, pos=0),
"field2": UInt32Col(shape=(), dflt=0, pos=1),
"field3": Float64Col(shape=(), dflt=0.0, pos=2),
"field4": Float32Col(shape=(), dflt=0.0, pos=3),
"field5": Float32Col(shape=(), dflt=0.0, pos=4),
"field6": Float32Col(shape=(), dflt=0.0, pos=5),
"field7": Float32Col(shape=(), dflt=0.0, pos=6),
"field8": Float32Col(shape=(), dflt=0.0, pos=7),
"field9": Float32Col(shape=(), dflt=0.0, pos=8),
"field10": Float32Col(shape=(), dflt=0.0, pos=9),
"field11": UInt16Col(shape=(), dflt=0, pos=10),
"field12": UInt16Col(shape=(), dflt=0, pos=11),
"field13": UInt16Col(shape=(), dflt=0, pos=12),
"field14": UInt16Col(shape=(), dflt=0, pos=13),
"field15": Float64Col(shape=(), dflt=0.0, pos=14),
"field16": Float32Col(shape=(), dflt=0.0, pos=15),
"field17": UInt16Col(shape=(), dflt=0, pos=16)}
byteorder := 'little'
chunkshape := (248,)
when I run a query on it this is the result:
>>> start=time.time(); data=f.root.table.readWhere('field1==2912'); print
>>> time.time()-start
11.0780000687
>>> len(data)
20678
I wanted to speed up this sort of querying, so created a new table with blosc
compression and copied the data. My old table has expectedrows = 1000000, but
since reality turned out to be a lot more data I also updated expectedrows to
10000000
/table1 (Table(17801755,), shuffle, blosc(1)) ''
description := {
"field1": UInt32Col(shape=(), dflt=0, pos=0),
"field2": UInt32Col(shape=(), dflt=0, pos=1),
"field3": Float64Col(shape=(), dflt=0.0, pos=2),
"field4": Float32Col(shape=(), dflt=0.0, pos=3),
"field5": Float32Col(shape=(), dflt=0.0, pos=4),
"field6": Float32Col(shape=(), dflt=0.0, pos=5),
"field7": Float32Col(shape=(), dflt=0.0, pos=6),
"field8": Float32Col(shape=(), dflt=0.0, pos=7),
"field9": Float32Col(shape=(), dflt=0.0, pos=8),
"field10": Float32Col(shape=(), dflt=0.0, pos=9),
"field11": UInt16Col(shape=(), dflt=0, pos=10),
"field12": UInt16Col(shape=(), dflt=0, pos=11),
"field13": UInt16Col(shape=(), dflt=0, pos=12),
"field14": UInt16Col(shape=(), dflt=0, pos=13),
"field15": Float64Col(shape=(), dflt=0.0, pos=14),
"field16": Float32Col(shape=(), dflt=0.0, pos=15),
"field17": UInt16Col(shape=(), dflt=0, pos=16)}
byteorder := 'little'
chunkshape := (3971,)
>>> start=time.time(); data=f.root.table1.readWhere('field1==2912'); print
>>> time.time()-start
115.51699996
>>> len(data)
20678
Not exactly what I expected! I am obviously doing something wrong. Any
suggestions?
Thanks, Koert
------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
