Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

2012-07-16 Thread Jacob Bennett
Wait, is there perhaps a way to simulataneously read and write without any kind of blocking? Perhaps the a mode or the r+ mode might help for simultaneous read/write? I am currently implementing the multithreading.Queue, but I think that a large number of query requests might put an necessary load

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

2012-07-14 Thread Antonio Valentino
Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

2012-07-14 Thread Jacob Bennett
Awesome, I think this sounds like a very workable solution and the idea is very neat. I will try to implement this right away. I definitely agree to putting a small example. Let you know how this works, thanks guys! Thanks, Jacob On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

2012-07-14 Thread Anthony Scopatz
+1 to example of this! On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: Awesome, I think this sounds like a very workable solution and the idea is very neat. I will try to implement this right away. I definitely agree to putting a small example. Let you know

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

2012-07-13 Thread Anthony Scopatz
On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

2012-07-13 Thread Josh Ayers
My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data