Hi Lionel,
Consistent, atomic, file i/o is sort of a fundamentally serial task.
Trying to do this in parallel is almost guaranteed to fail in one
way or another.
What you need is a caching / blocking mechanism on top of the
HDF5 file. All of your processes would write to this queue which
would then write to the table when it gets the spare cycles.
It wouldn't be too hard to do. I would look into ZeroMQ and pyzmq.
Perhaps other people have other suggestions...
Be Well
Anthony
On Wed, May 25, 2011 at 4:39 AM, lionel chiron <lionel.chi...@gmail.com>wrote:
> Hi All,
>
> we tried to use pytables with multiprocessing (multiprocessing module).
> When reading various rows of a hdf5 file in various process there is no
> problem but if we to write the result of the row calculus in an other hdf5
> file it crashes unexpectedily.
> If not using multiprocessing there is no issue. Did someone had the same
> problem by reading and writing in DIFFERENT files?
> Thanks!
>
> Cheers
> Lionel
>
>
> ------------------------------------------------------------------------------
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery,
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now.
http://p.sf.net/sfu/quest-d2dcopy1
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
