A Dijous 23 Març 2006 19:29, Andrew Straw va escriure:
> Francesc Altet wrote:
> >Another thing can be to call File.flush() regularly, specially after
> >doing a writing operation. Be careful, because this may drastically
> >reduce the writing performance.
>
> Francesc, can you elaborate on "drastically reduce"? In my case, I'm
> already writing dealing with PyTables from a separate thread, exactly so
> that I don't care too much how long an operation takes.
Well, by forcing flushing too frequently you are not taken advantage
of the cache system (the PyTables one, but also the HDF5 and the
filesystem ones) at all. As a simple example, the next code:
------------------------------------------------------------
from tables import *
class Test(IsDescription):
col1 = IntCol()
col2 = FloatCol(shape=(3,4))
col3 = StringCol(length=3)
f = openFile("/tmp/test.h5", "w")
table = f.createTable(f.root, "table", Test, filters=Filters(complevel=1))
for i in xrange(1000*10):
table.row.append() # append a row with default values
##f.flush() # Uncomment for flushing!!!!
f.close()
------------------------------------------------------------
takes 0m0.678s in my laptop. The same script, but uncommenting the
flushing line takes 1m25.622s. So, you have to be careful with
flushing too much (unless performance is not important for your app).
>
> Also, in a rather similar email thread a couple weeks ago (titled
> "corrupt HDF5 files"), Ivan suggested, "you could try to explicitly
> close each of the leaves you have open (without closing the file
> itself)". What methods would I call to do this? Is this File.flush() or
> is there something more?
Yes, for closing nodes there is the method Node._f_close() (or
Leaf.close() which is equivalent). Closing a node is a good way to
ensure that you force a flush on data belonging to this node (and only
that), so reducing the risk.
> I already call Table.flush() whenever I'm done adding rows.
That's the normal thing to do: write a bunch of rows and do a flush
after this. This will minimize the probability of ending with a
corrupted file in case of a program bomb out.
Hope that helps,
--
>0,0< Francesc Altet http://www.carabos.com/
V V Cárabos Coop. V. Enjoy Data
"-"
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
