Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
Wait, is there perhaps a way to simulataneously read and write without any kind of blocking? Perhaps the a mode or the r+ mode might help for simultaneous read/write? I am currently implementing the multithreading.Queue, but I think that a large number of query requests might put an necessary load on my writing queue since the data comes in so fast. ;) Btw, I will submit the example soon. -Jacob On Sat, Jul 14, 2012 at 1:39 PM, Anthony Scopatz scop...@gmail.com wrote: +1 to example of this! On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: Awesome, I think this sounds like a very workable solution and the idea is very neat. I will try to implement this right away. I definitely agree to putting a small example. Let you know how this works, thanks guys! Thanks, Jacob On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data read from the file could be sent back to the originating process using another queue or pipe. You should be able to do the same thing with sockets if the other parts of your application are in languages other than Python. I do something similar to handle writing to a log file from multiple processes and it works well. In that case the file is write-only - and just a simple text file rather than HDF5 - but I don't see any reason why it wouldn't work for read and write as well. Hope that helps, Josh I totally agree with Josh. I don't have a test code to demonstrate it but IMHO parallelizing I/O to/from a single file on a single disk do not makes too much sense unless you have special HW. Is this your case Jacob? IMHO with standard SATA devices you could have a marginal speedup (in the best case), but if your bottleneck is the I/O this will not solve your problem. If someone finds the time to implement a toy example of what Josh suggested we could put it on the cookbook :) regards On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz scop...@gmail.com wrote: On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared over separate processes, only threads. So could you give me any advice in this situation? I'm sure it has come up before. ;) Hello All, I previously suggested to Jacob a setup where only one proc would have a write handle and all of the other processes would be in read-only mode. I am not sure that this would work. Francesc, Antonio, Josh, etc or anyone else, how would you solve this problem where you may want many processors to query the file, while something else may be writing to it? I defer to people with more experience... Thanks for your help! Be Well Anthony Thanks, Jacob Bennett -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile
Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data read from the file could be sent back to the originating process using another queue or pipe. You should be able to do the same thing with sockets if the other parts of your application are in languages other than Python. I do something similar to handle writing to a log file from multiple processes and it works well. In that case the file is write-only - and just a simple text file rather than HDF5 - but I don't see any reason why it wouldn't work for read and write as well. Hope that helps, Josh I totally agree with Josh. I don't have a test code to demonstrate it but IMHO parallelizing I/O to/from a single file on a single disk do not makes too much sense unless you have special HW. Is this your case Jacob? IMHO with standard SATA devices you could have a marginal speedup (in the best case), but if your bottleneck is the I/O this will not solve your problem. If someone finds the time to implement a toy example of what Josh suggested we could put it on the cookbook :) regards On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz scop...@gmail.com wrote: On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared over separate processes, only threads. So could you give me any advice in this situation? I'm sure it has come up before. ;) Hello All, I previously suggested to Jacob a setup where only one proc would have a write handle and all of the other processes would be in read-only mode. I am not sure that this would work. Francesc, Antonio, Josh, etc or anyone else, how would you solve this problem where you may want many processors to query the file, while something else may be writing to it? I defer to people with more experience... Thanks for your help! Be Well Anthony Thanks, Jacob Bennett -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
Awesome, I think this sounds like a very workable solution and the idea is very neat. I will try to implement this right away. I definitely agree to putting a small example. Let you know how this works, thanks guys! Thanks, Jacob On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data read from the file could be sent back to the originating process using another queue or pipe. You should be able to do the same thing with sockets if the other parts of your application are in languages other than Python. I do something similar to handle writing to a log file from multiple processes and it works well. In that case the file is write-only - and just a simple text file rather than HDF5 - but I don't see any reason why it wouldn't work for read and write as well. Hope that helps, Josh I totally agree with Josh. I don't have a test code to demonstrate it but IMHO parallelizing I/O to/from a single file on a single disk do not makes too much sense unless you have special HW. Is this your case Jacob? IMHO with standard SATA devices you could have a marginal speedup (in the best case), but if your bottleneck is the I/O this will not solve your problem. If someone finds the time to implement a toy example of what Josh suggested we could put it on the cookbook :) regards On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz scop...@gmail.com wrote: On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared over separate processes, only threads. So could you give me any advice in this situation? I'm sure it has come up before. ;) Hello All, I previously suggested to Jacob a setup where only one proc would have a write handle and all of the other processes would be in read-only mode. I am not sure that this would work. Francesc, Antonio, Josh, etc or anyone else, how would you solve this problem where you may want many processors to query the file, while something else may be writing to it? I defer to people with more experience... Thanks for your help! Be Well Anthony Thanks, Jacob Bennett -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
+1 to example of this! On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: Awesome, I think this sounds like a very workable solution and the idea is very neat. I will try to implement this right away. I definitely agree to putting a small example. Let you know how this works, thanks guys! Thanks, Jacob On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data read from the file could be sent back to the originating process using another queue or pipe. You should be able to do the same thing with sockets if the other parts of your application are in languages other than Python. I do something similar to handle writing to a log file from multiple processes and it works well. In that case the file is write-only - and just a simple text file rather than HDF5 - but I don't see any reason why it wouldn't work for read and write as well. Hope that helps, Josh I totally agree with Josh. I don't have a test code to demonstrate it but IMHO parallelizing I/O to/from a single file on a single disk do not makes too much sense unless you have special HW. Is this your case Jacob? IMHO with standard SATA devices you could have a marginal speedup (in the best case), but if your bottleneck is the I/O this will not solve your problem. If someone finds the time to implement a toy example of what Josh suggested we could put it on the cookbook :) regards On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz scop...@gmail.com wrote: On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared over separate processes, only threads. So could you give me any advice in this situation? I'm sure it has come up before. ;) Hello All, I previously suggested to Jacob a setup where only one proc would have a write handle and all of the other processes would be in read-only mode. I am not sure that this would work. Francesc, Antonio, Josh, etc or anyone else, how would you solve this problem where you may want many processors to query the file, while something else may be writing to it? I defer to people with more experience... Thanks for your help! Be Well Anthony Thanks, Jacob Bennett -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared over separate processes, only threads. So could you give me any advice in this situation? I'm sure it has come up before. ;) Hello All, I previously suggested to Jacob a setup where only one proc would have a write handle and all of the other processes would be in read-only mode. I am not sure that this would work. Francesc, Antonio, Josh, etc or anyone else, how would you solve this problem where you may want many processors to query the file, while something else may be writing to it? I defer to people with more experience... Thanks for your help! Be Well Anthony Thanks, Jacob Bennett -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data read from the file could be sent back to the originating process using another queue or pipe. You should be able to do the same thing with sockets if the other parts of your application are in languages other than Python. I do something similar to handle writing to a log file from multiple processes and it works well. In that case the file is write-only - and just a simple text file rather than HDF5 - but I don't see any reason why it wouldn't work for read and write as well. Hope that helps, Josh On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz scop...@gmail.com wrote: On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared over separate processes, only threads. So could you give me any advice in this situation? I'm sure it has come up before. ;) Hello All, I previously suggested to Jacob a setup where only one proc would have a write handle and all of the other processes would be in read-only mode. I am not sure that this would work. Francesc, Antonio, Josh, etc or anyone else, how would you solve this problem where you may want many processors to query the file, while something else may be writing to it? I defer to people with more experience... Thanks for your help! Be Well Anthony Thanks, Jacob Bennett -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
