Hmm, the specific test I wrote (see below) fails also in pytables 1.1.1,
but my memory is that my application was usually saved self-consistent
files when it was unexpectedly terminated (which included Ctrl-C back
then). So, I'm not sure exactly what's changed, but I can't reproduce
it. I'm prepared to live with my current 'solution' -- closing and
re-opening the file every time I save to it. It's small price to pay for
greatly decreasing the chances of losing my data.
Here's the test code I'm using. I kill the code during the sleep
statement and then check the hdf5 file with 'h5ls' or similar utility.
I will try with the latest HDF5 one day (not soon, probably) and let you
know what the results are.
import tables
import time
class Info2D(tables.IsDescription):
colA = tables.Int32Col(pos=0)
colB = tables.Int32Col(pos=1)
h5file = tables.openFile('test.h5',mode='w')
table = h5file.createTable(h5file.root,'data2d', Info2D, "2d data")
for i in range(6000):
table.row['colA'] = i+10
table.row['colB'] = i
table.row.append()
table.flush()
table.close()
del table
loop_times = []
for j in range(10):
print j,'sleep start'
time.sleep(2)
print j,'sleep stop'
t1 = time.time()
table = h5file.root.data2d
for i in range(6000):
table.row['colA'] = j*10000 + i+10
table.row['colB'] = j*10000 + i
table.row.append()
table.flush()
table.close()
del table
# make sure everything is flushed
#h5file.close()
#h5file = tables.openFile('test.h5',mode='r+')
t2 = time.time()
dur = (t2-t1)*1e3
loop_times.append(dur)
print loop_times
Francesc Altet wrote:
Hi Andrew,
A Dilluns 10 Abril 2006 20:39, Andrew Straw va escriure:
Just for everyone's information:
I've finally had a chance to play around with this a little bit, and
I've found the only thing that makes my hdf5 files resistant to my
program being terminated (e.g. with "kill -9" while in the middle of a
time.sleep() call). Is to actually close and re-open the file after each
write. The suggestion to call Leaf.close() (== Table.close()) after
appending rows doesn't fix the hdf5 integrity issues. I suspect this may
be because I'm not calling close() (or _f_close()) on the root group. I
haven't tried doing that because I'm not sure what other effects that
would have.
This is intersting to know. In order to look for a reason for this to
fail, it would be nice to know if PyTables 1.1.1 would be more stable
in that regard than 1.2 (or 1.3). The reason for this can be the
introduction of the LRU cache in PyTables 1.2. The LRU cache allows to
fix a limit of open leaves in-memory, and this is why we decided to
not open/close the leaves on every read/write operation (the number of
open handles for disk objects is limited). This is supposed to
accelerate the I/O, but of course might raise the probability to spoil
the file.
So, if you can experiment with PyTables 1.1.1 and your files are more
stable, then we've found a possible reason of the problem. It also
would help if you try with latest stable HDF5 (1.6.5), in order to
avoid problems on the HDF5 side.
Thanks for any feedback,
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
