A Thursday 30 July 2009 12:22:27 David Fokkema escrigué: > On Wed, 2009-07-29 at 18:47 +0200, Francesc Alted wrote: > > Hi David, > > > > If you need concurrency for writing you can always setup a data collector > > that gathers info from the several threads by using the `Queue.queue()` > > container. As it is thread safe, you don't have to worry about > > concurrency problems. > > Yes, I've used that before quite succesfully. I'm however not quite sure > how separate apache threads will be able to contact my writer thread. > Maybe I'll have to dive into apache documentation. Another problem is > that we'd like to have a confirmation that the data was actually stored, > so simple queuing won't suffice, I guess. Using threads and loading off > writing data to one thread is a solution, then.
This exactly what I meant. Several threads sending data to the same queue that will be extracted by the writing thread later on. You can also arbitrate a simple protocol for ensuring that data has been effectively written (or not). [clip] > > My preferences here go to use a monolithic table to save your daily > > observations, and then make use of the indexing capabilities of PyTables > > Pro for locating and retrieving your data quickly. If you can't afford > > buying > > We'll take that into consideration. My personal preference would be to > always use open source, so pro is in that respect a step back. However, > I'd rather spend money on the pro version of a product I really like > with a very dedicated maintainer than on some other proprietary > solution. We basically don't have much money to spend because we're a > relatively small outreach project which lives off donations from our > institute and some industrial partners (earmarked for particular schools > in their neighborhood and not to be spent centrally). > > Just curious, how would you look upon someone else implementing OPSI in > pytables (fully open source)? Mmh, that's a good question :-) Well, provided that Pro allows me to dedicate time to the PyTables project as a whole, I must admit that I won't be very happy if that happens one day, and perhaps some users won't be happy neither that the main developer of PyTables has to cease his work on it --unless of course, other developers can continue my work by using another business model. Having said this, I can't stop nobody doing things, and most specially try to prevent implementing new indexing schemes on top of PyTables, which can be a *good* thing for the community. However, re-implementing OPSI in open-source seems to me like a very little innovative project for others, so it might be more interesting to go for something better (which is possible for sure). > > Pro, perhaps you can split your data in several tables (hourly tables?), > > setup some search code that can select the appropriate table and then do > > an in- kernel query for the interesting table. > > Several identical (in structure) tables in one file? Haven't thought of > that... Why not? It can be a way to have better access to your data (if properly categorized, of course). -- Francesc Alted ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
