Hi Mario,
A Sunday 26 September 2010 04:34:00 Mario Juric escrigué:
> Hi everyone,
> I'm evaluating PyTables as a backend for data storage in large
> astronomical surveys. So far the results have been fantastic, except
> for the following issue: if a table is opened and iterated on from
> within a subprocess instantiated with multiprocessing.Process, the
> program hangs.
You are trying to use different where iterators from two processes and
this is not supported (to start with, you need to compile HDF5 in
thread-safe mode).
However, you may want to pass the same iterator to the different
processes, and extract elements in turn. This should work because the
generator is held by only one process. Here it is what I mean:
import tables
import multiprocessing as mp
def _worker(rows, qout = None):
print "Got the iterator, about to iterate"
row = next(rows)
print "Succeeded in one iteration\n"
if qout is not None:
qout.put("Done")
if __name__ == "__main__":
fn = "testcase.h5"
fp = tables.openFile(fn)
print "About to load: ", fn
rows = fp.root.catalog.where('(ra > 0)')
print "**** Running from main process:"
_worker(rows)
print "**** Running from subprocess:"
qout = mp.Queue()
ps = mp.Process(target=_worker, args=(rows, qout))
ps.daemon = True
ps.start()
print qout.get();
fp.close()
which runs perfect. Would that work for you?
--
Francesc Alted
------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
