Hi Antony,
thank you for your answer and propositions to debug the issue.
We knew there could be problems with i/o with hdf5 but what we do seems
consistent with a caching / blocking procedure..
The way it is done is :
1)to realize a pool..
2)to associate a calculus to a function and iterator (index on columns of
hdf5 first file).. with pool.imap
3)to make a loop on the imap with writing each result in the hdf5 (second
file)..
The loop allows in principle the calculus to be called one after an other..
in a way it should be a blocking maneer to fill our matrix.. no?
Best
Lionel
2011/5/25 Anthony Scopatz <scop...@gmail.com>
> Hi Lionel,
>
> Consistent, atomic, file i/o is sort of a fundamentally serial task.
> Trying to do this in parallel is almost guaranteed to fail in one
> way or another.
>
> What you need is a caching / blocking mechanism on top of the
> HDF5 file. All of your processes would write to this queue which
> would then write to the table when it gets the spare cycles.
>
> It wouldn't be too hard to do. I would look into ZeroMQ and pyzmq.
>
> Perhaps other people have other suggestions...
>
> Be Well
> Anthony
>
> On Wed, May 25, 2011 at 4:39 AM, lionel chiron <lionel.chi...@gmail.com>wrote:
>
>> Hi All,
>>
>> we tried to use pytables with multiprocessing (multiprocessing module).
>> When reading various rows of a hdf5 file in various process there is no
>> problem but if we to write the result of the row calculus in an other hdf5
>> file it crashes unexpectedily.
>> If not using multiprocessing there is no issue. Did someone had the same
>> problem by reading and writing in DIFFERENT files?
>> Thanks!
>>
>> Cheers
>> Lionel
>>
>>
>> ------------------------------------------------------------------------------
>> vRanger cuts backup time in half-while increasing security.
>> With the market-leading solution for virtual backup and recovery,
>> you get blazing-fast, flexible, and affordable data protection.
>> Download your free trial now.
>> http://p.sf.net/sfu/quest-d2dcopy1
>> _______________________________________________
>> Pytables-users mailing list
>> Pytables-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery,
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now.
http://p.sf.net/sfu/quest-d2dcopy1
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
