En/na Andrew Straw ha escrit:: >[...] > On another note, it seems like part of the issue is that I was used to > an older version of pytables (I think from 20050715 or so) in which the > .h5 files seemed to be in an internally consistent state 99+% of the > time, even while the app had them open. For example, I could make a copy > of the still-growing .h5 file and open it with my analysis tools, and I > never had the corrupt file issue. (Or do a simple h5ls on the file.) > Now, however, with the 20060306 snapshot, I haven't once been able to do > this. Can you think of anything different in pytables that may lead to > this behavior? Is there anything I can do in my application like call > h5file.sync() or something to attempt to force the file to be internally > consistent? >[...]
I wouldn't dare to say that copying an open HDF5 file using an external
process should be safe, but I still think I could explain the reason why
you had better luck with the old version. Before PyTables 1.2, leaves
got closed as soon as read and write operations finished, so as to keep
memory consumption low--remember that PyTables kept *all* nodes in
memory. From 1.2 on, only a small amount of nodes remain in memory
(thanks to the LRU cache), and they are kept *open* to reduce the
overhead of opening and closing them each time they are accessed. This
is more space and time-efficient, but keeping the nodes open may mean
that the HDF5 file remains in a consistent state less time.
As a simple test, you could try to explicitly close each of the leaves
you have open (without closing the file itself), and then try to copy
the file and check if it is OK. If this is the case, then maybe my
suspicion is right.
Anyway, I would still not recommend copying an open file like that. If
you have any ideas to make the situation better, they would be most welcome.
Regards,
::
Ivan Vilata i Balaguer >qo< http://www.carabos.com/
Cárabos Coop. V. V V Enjoy Data
""
signature.asc
Description: OpenPGP digital signature
