Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: > My first instinct would be to handle all access (read and write) to > that file from a single process. You could create two > multiprocessing.Queue objects, one for data to write and one for read > requests. Then the process would check the queues in a loop and > handle each request serially. The data read from the file could be > sent back to the originating process using another queue or pipe. You > should be able to do the same thing with sockets if the other parts of > your application are in languages other than Python. > > I do something similar to handle writing to a log file from multiple > processes and it works well. In that case the file is write-only - > and just a simple text file rather than HDF5 - but I don't see any > reason why it wouldn't work for read and write as well. > > Hope that helps, > Josh >
I totally agree with Josh. I don't have a test code to demonstrate it but IMHO parallelizing I/O to/from a single file on a single disk do not makes too much sense unless you have special HW. Is this your case Jacob? IMHO with standard SATA devices you could have a marginal speedup (in the best case), but if your bottleneck is the I/O this will not solve your problem. If someone finds the time to implement a toy example of what Josh suggested we could put it on the cookbook :) regards > On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <scop...@gmail.com> wrote: >> On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett <jacob.bennet...@gmail.com> >> wrote: >> >> [snip] >> >>> >>> My first implementation was to have a set of current files stay in write >>> mode and have an overall lock over these files for the current day, but >>> (stupidly) I forgot that lock instances cannot be shared over separate >>> processes, only threads. >>> >>> So could you give me any advice in this situation? I'm sure it has come up >>> before. ;) >> >> >> Hello All, I previously suggested to Jacob a setup where only one proc would >> have a write handle and all of the other processes would be in read-only >> mode. I am not sure that this would work. >> >> Francesc, Antonio, Josh, etc or anyone else, how would you solve this >> problem where you may want many processors to query the file, while >> something else may be writing to it? I defer to people with more >> experience... Thanks for your help! >> >> Be Well >> Anthony >> >>> >>> Thanks, >>> Jacob Bennett >>> -- Antonio Valentino ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
