I'll second the use of the multiprocessing library. I started out using cron to start up a couple independent processes, but quickly ran into issues depending on the complexity of what I was trying to do. In the end I settled on the mp library which has worked out well.
On May 23, 4:58 am, Michele Comitini <[email protected]> wrote: > I suggest to use the multiprocessing > modulehttp://docs.python.org/library/multiprocessing.html > and avoid threads. It is standard in python 2.6+ and you can find > backports to 2.5 also. > API is similar to Threading. > > mic > > 2011/5/23 Iceberg <[email protected]>: > > > > > > > > > On May 20, 9:40 pm, Ross Peoples <[email protected]> wrote: > >> If I am reading this properly, you want to create a new instance if db for > >> each thread? I have never tried this before, but could you do something > >> like > >> this for each thread? > > >> db_thread = DAL(db._uri) > >> for k, v in db.items(): > >> db_thread[k] = v > > >> The idea being to using the existing db's URI string to make a new DAL > >> instance, then copying the table information from db to db_thread. Again, I > >> don't know if this works, but it might point in the right direction unless > >> someone else has a more elegant solution. > > > Or another direction. DB is generally not designed to handle > > concurrency very well. So maybe Kimmo can have his script handle > > concurrency by gathering data into a queue, then use a 5th thread to > > dump data into db. > > > Regards, > > Iceberg
