On Wed, 2011-03-09 at 05:43 +1100, Tony Theodore wrote: > On 9 March 2011 03:55, Felipe Barriga Richards <fbarr...@algometrics.cl> > wrote: > > On Tue, 2011-03-08 at 03:21 +1100, Tony Theodore wrote: > >> Hi Felipe, > >> > >> This is curious, I'm guessing that you're opening the file once, then > >> using multiple processes to access it. Can you send the lines before > >> line 49, or a subset that reproduces this behaviour? > >> > >> > When I try the same but using copies of the file (not symbolic links) it > >> > work fine. > >> > >> Well, copies are distinct files, so you wouldn't expect there to be a > >> problem. I'm guessing again, but this is probably only accidental - if > >> you loaded up more processes it would fail in the same way. > > > > Tony, > > > > I got the bug ! I was opening the file on the process constructor. When > > I move it to the "run" method it works without problems. > > Weird ? > > Glad you sorted it out, thanks for reporting back. I haven't really > used the multiprocessing module before, but it's good to know it works > as I plan on doing something similar in the near future. > > Cheers, > > Tony
The only problem that I've found is that a Queue is quite slow. If you have multiple process reading HDF5 files and then sending the results to a common queue is extremely slow (actually is faster to load each HDF5 file in 1 process rather than using all CPU and then store the result in a Queue). Maybe using ZMQ to communicate processes (or to a single 'master' process) could help. Regards, -- Felipe Barriga Richards Algometrics SpA ------------------------------------------------------------------------------ What You Don't Know About Data Connectivity CAN Hurt You This paper provides an overview of data connectivity, details its effect on application quality, and explores various alternative solutions. http://p.sf.net/sfu/progress-d2d _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
