Hi have difficutties with my mail on my main computer where is my code so I
can't esaily paste my code here.
But what you sugggest about flushing and closing could be a good reason for
why it doesn't work.. I'll try this way (that is the simplest one to test
first)
Cheers
Lionel
2011/5/25 Anthony Scopatz <scop...@gmail.com>
> Hmm without knowing the code base at all, it seems like there could be
> other issues.
> Here are a few ideas:
>
> 1) Are you making sure to flush() or close() the file after each write?
> 2) It may be safest to each write event to also have its own, new file
> handler (ie call openFile again)
> 3) Have a separate process that is the only process that is allowed to
> touch the file
>
> Be Well
> Anthony
>
>
> On Wed, May 25, 2011 at 11:52 AM, lionel chiron
> <lionel.chi...@gmail.com>wrote:
>
>> Hi Antony,
>> thank you for your answer and propositions to debug the issue.
>> We knew there could be problems with i/o with hdf5 but what we do seems
>> consistent with a caching / blocking procedure..
>> The way it is done is :
>> 1)to realize a pool..
>> 2)to associate a calculus to a function and iterator (index on columns of
>> hdf5 first file).. with pool.imap
>> 3)to make a loop on the imap with writing each result in the hdf5 (second
>> file)..
>> The loop allows in principle the calculus to be called one after an
>> other..
>> in a way it should be a blocking maneer to fill our matrix.. no?
>>
>> Best
>> Lionel
>>
>>
>>
>> 2011/5/25 Anthony Scopatz <scop...@gmail.com>
>>
>>> Hi Lionel,
>>>
>>> Consistent, atomic, file i/o is sort of a fundamentally serial task.
>>> Trying to do this in parallel is almost guaranteed to fail in one
>>> way or another.
>>>
>>> What you need is a caching / blocking mechanism on top of the
>>> HDF5 file. All of your processes would write to this queue which
>>> would then write to the table when it gets the spare cycles.
>>>
>>> It wouldn't be too hard to do. I would look into ZeroMQ and pyzmq.
>>>
>>> Perhaps other people have other suggestions...
>>>
>>> Be Well
>>> Anthony
>>>
>>> On Wed, May 25, 2011 at 4:39 AM, lionel chiron
>>> <lionel.chi...@gmail.com>wrote:
>>>
>>>> Hi All,
>>>>
>>>> we tried to use pytables with multiprocessing (multiprocessing module).
>>>> When reading various rows of a hdf5 file in various process there is no
>>>> problem but if we to write the result of the row calculus in an other hdf5
>>>> file it crashes unexpectedily.
>>>> If not using multiprocessing there is no issue. Did someone had the same
>>>> problem by reading and writing in DIFFERENT files?
>>>> Thanks!
>>>>
>>>> Cheers
>>>> Lionel
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> vRanger cuts backup time in half-while increasing security.
>>>> With the market-leading solution for virtual backup and recovery,
>>>> you get blazing-fast, flexible, and affordable data protection.
>>>> Download your free trial now.
>>>> http://p.sf.net/sfu/quest-d2dcopy1
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pytables-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> vRanger cuts backup time in half-while increasing security.
>>> With the market-leading solution for virtual backup and recovery,
>>> you get blazing-fast, flexible, and affordable data protection.
>>> Download your free trial now.
>>> http://p.sf.net/sfu/quest-d2dcopy1
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pytables-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> vRanger cuts backup time in half-while increasing security.
>> With the market-leading solution for virtual backup and recovery,
>> you get blazing-fast, flexible, and affordable data protection.
>> Download your free trial now.
>> http://p.sf.net/sfu/quest-d2dcopy1
>> _______________________________________________
>> Pytables-users mailing list
>> Pytables-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
vRanger cuts backup time in half-while increasing security.
With the market-leading solution for virtual backup and recovery,
you get blazing-fast, flexible, and affordable data protection.
Download your free trial now.
http://p.sf.net/sfu/quest-d2dcopy1
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
