Hi Everyone,
I am testing the I/O speed on my machine.
I got a array in HDF5 file:
File(xxx.h5, title='', mode='r', rootUEP='/',
filters=Filters(complevel=0, shuffle=False, fletcher32=False))
/ (RootGroup) ''
/rawSignal (Array(3000000, 24)) ''
atom := Float64Atom(shape=(), dflt=0.0)
maindim := 0
flavor := 'numpy'
byteorder := 'big'
chunkshape := None
h5file=openFile("xxx.h5", mode = "r")
a1=h5file.root.rawSignal
then I try to load some chunks into a numpy array:
import numpy as npy
def test(a_len):
a=npy.zeros((a_len,1))
t1=time.time()
a[:,0]=a1[:a_len,1]
t2=time.time()-t1
print 'a size = ',a.shape,'dt = ',t2
Then I got following results:
data lenght = 100.0
a size = (100L, 1L) dt = 0.0
data lenght = 1000.0
a size = (1000L, 1L) dt = 0.0310001373291
data lenght = 10000.0
a size = (10000L, 1L) dt = 0.281000137329
data lenght = 100000.0
a size = (100000L, 1L) dt = 2.70299983025
data lenght = 1000000.0
a size = (1000000L, 1L) dt = 27.0629999638
another file:
File(filename=yyy.h5, title='', mode='r', rootUEP='/',
filters=Filters(complevel=0, shuffle=False, fletcher32=False))
/ (RootGroup) ''
/rawSignal (CArray(3000000, 24)) ''
atom := Float64Atom(shape=(), dflt=0.0)
maindim := 0
flavor := 'numpy'
byteorder := 'little'
chunkshape := (1365, 24)
a size = (100L, 1L) dt = 0.0
a size = (1000L, 1L) dt = 0.0
a size = (10000L, 1L) dt = 0.0160000324249
a size = (100000L, 1L) dt = 0.0149998664856
a size = (1000000L, 1L) dt = 0.31200003624
Is this speed normal? So the major difference is due to the chunkshape I
guess. But how could I specify the chunckshape when I open a file
without a chunkshape infomation?
------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
