A Thursday 13 January 2011 16:37:35 Ben Kraft escrigué:
[clip]
> The test works for me if the where condition only returns a small
> number of rows (~40), but fails for larger queries (14000 rows). In
> these cases, the query will will finish on the main process, but the
> forked process hangs after iterating through about half the total
> rows on the fork. Admittedly, I don't know much about
> multiprocessing in python, and this hang might not be caused by
> pytables.
Hmm, I'm sorry, but I cannot reproduce the problem, at least with a
truly self-contained example (attached), and for any number of rows
returned (tried until 1 million). Could you make a variation of this
script to make it expose the bad behaviour?
> Relevant versions,
>
> tables.__version__
> Out[2]: '2.2pro'
Any reason you are not running 2.2.1?
> numexpr.__version__
> Out[4]: '1.4.1'
>
> and python 2.6.6.
These look fine. The complete output for `tables.print_versions()`
would help too.
--
Francesc Alted
import tables
import multiprocessing as mp
filename = "/tmp/mp-bug.h5"
NROWS = 1000*1000
def create_file():
f = tables.openFile(filename, "w")
td = {"origWac":tables.Float32Col(), "other":tables.Int32Col()}
t = f.createTable(f.root, 't', td)
r = t.row
for i in xrange(NROWS):
r["origWac"] = i
r["other"] = i*2
r.append()
f.close()
def _worker(qout = None):
fp = tables.openFile(filename)
print "About to load: ", filename
rows = fp.root.t.where('(origWac >5.9) & (origWac<58000)')
count = 0
for row in rows:
count +=1
print count,
print
fp.close()
if qout is not None:
qout.put("Done")
if __name__ == "__main__":
print "**** Creating an input file:"
create_file()
print "**** Running from main process:"
_worker()
print "**** Running from subprocess:"
qout = mp.Queue()
ps = mp.Process(target=_worker, args=(qout,))
ps.daemon = True
ps.start()
print qout.get()
------------------------------------------------------------------------------
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand
malware threats, the impact they can have on your business, and how you
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
