My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data read from the file could be sent back to the originating process using another queue or pipe. You should be able to do the same thing with sockets if the other parts of your application are in languages other than Python.
I do something similar to handle writing to a log file from multiple processes and it works well. In that case the file is write-only - and just a simple text file rather than HDF5 - but I don't see any reason why it wouldn't work for read and write as well. Hope that helps, Josh On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz <scop...@gmail.com> wrote: > On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett <jacob.bennet...@gmail.com> > wrote: > > [snip] > >> >> My first implementation was to have a set of current files stay in write >> mode and have an overall lock over these files for the current day, but >> (stupidly) I forgot that lock instances cannot be shared over separate >> processes, only threads. >> >> So could you give me any advice in this situation? I'm sure it has come up >> before. ;) > > > Hello All, I previously suggested to Jacob a setup where only one proc would > have a write handle and all of the other processes would be in read-only > mode. I am not sure that this would work. > > Francesc, Antonio, Josh, etc or anyone else, how would you solve this > problem where you may want many processors to query the file, while > something else may be writing to it? I defer to people with more > experience... Thanks for your help! > > Be Well > Anthony > >> >> Thanks, >> Jacob Bennett >> >> -- >> Jacob Bennett >> Massachusetts Institute of Technology >> Department of Electrical Engineering and Computer Science >> Class of 2014| benne...@mit.edu >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Pytables-users mailing list >> Pytables-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/pytables-users >> > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Pytables-users mailing list > Pytables-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pytables-users > ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
