Hello Owen,
While you can use process pools to read from a file in parallel just fine,
writing is another story completely. While HDF5 itself supports parallel
writing though MPI, this comes at the high cost of compression no longer
being available and a much more complicated code base. So for the time
being, PyTables only supports the serial HDF5 library.
Therefore if you want to write to a file in parallel, you adopt a strategy
where you have one process which is responsible for all of the writing and
all other processes send their data to this process instead of writing to
file directly. This is a very effective way of accomplishing basically
what you need. In fact, we have an example to do just that [1]. (As a
side note: HDF5 may soon be adding an API for exactly this pattern because
it comes up so often.)
So if I were you, I would look at [1] and adopt it to my use case.
Be Well
Anthony
1.
https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py
On Fri, Oct 5, 2012 at 9:55 AM, Owen Mackwood
<owen.mackw...@bccn-berlin.de>wrote:
> Hello,
>
> I'm using a multiprocessing.Pool to parallelize a set of tasks which
> record their results into separate hdf5 files. Occasionally (less than 2%
> of the time) the worker process will hang. According to gdb, the problem
> occurs while opening the hdf5 file, when it attempts to obtain the
> associated mutex. Here's part of the backtrace:
>
> #0 0x00007fb2ceaa716c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1 0x00007fb2be61c215 in H5TS_mutex_lock () from /usr/lib/libhdf5.so.6
> #2 0x00007fb2be32bff0 in H5open () from /usr/lib/libhdf5.so.6
> #3 0x00007fb2b96226a4 in __pyx_pf_6tables_13hdf5Extension_4File__g_new
> (__pyx_v_self=0x7fb2b04867d0, __pyx_args=<value optimized out>,
> __pyx_kwds=<value optimized out>)
> at tables/hdf5Extension.c:2820
> #4 0x00000000004abf62 in ext_do_call (f=0x4cb2430, throwflag=<value
> optimized out>) at Python/ceval.c:4331
>
> Nothing else is trying to open this file, so can someone suggest why this
> is occurring? This is a very annoying problem as there is no way to recover
> from this error, and consequently the worker process is permanently
> occupied, which effectively removes one of my processors from the pool.
>
> Regards,
> Owen Mackwood
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
