Michael Bayer wrote: > why are you using twisted *with* threads ? didnt we all agree that was > sort of unnecessary ?
Twisted operates without needing to run *in* a thread by doing everything asynchronously. Every call made via the Twisted event loop must either return (synchronous) results very quickly or return an immediate reference to an eventual result (thus operating asynchronously). The objects that hold those immediate references are called "deferreds" in TWisted parlance. Twisted runs most all of its internal operations with appropriate chunking and use of select() to keep the asynchronous event loop humming along nicely, but it has no control over how other libraries do things. When an external library like SQLAlchemy presents a blocking call like select(...).execute.fetchall(), the only way to make that call "play nice" with Twisted by immediately returning a deferred to the eventual result is by having TWisted run it in a thread. It includes the deferToThread() function for that express purpose. > (this goes to my point that twisted is a pain in the butt...) Perhaps, but asynchronous code is a whole different way of thinking that some of us actually find cleaner and more intuitive than blocking + threads. What I'm trying to do with sAsync is put the "pain in the butt" associated with making synchronous and asynchronous code work together firmly behind the scenes for Twisted users, at least as far as interacting with SQLAlchemy is concerned. > anyway, SA 0.2 uses the same idea for sqlite threading as 0.1....it > maintains each connection thread locally since you cant share a sqlite > connection between threads. im also not sure what youre talking about > with a "thread local copy of the engine", if thats something youre doing > on your end, you might want to not do that. use Connection objects > instead theyre much more portable. Mike, I remember you telling me a while ago that SA somehow figures out how to make things work with the engine. That was after I spent a lot of time trying to re-engineer the wheel, to great frustration on my part. I followed your advice, ditched my thread-local code, and got SA transactions just fine (based on the engine) to run in arbitrary threads. Now it's not working in SA 0.2. Could you point out what's changed in regard to the engine vs. threads vs. sqlite? I'd rather not change everything under the hood (e.g., using connections somehow) if I don't have to. The lesson of not trying to re-engineer what SA handled fine on its own is still with me. :-) Best regards, Ed > On May 27, 2006, at 6:57 PM, Ed Suominen wrote: > >> My sAsync project relies on Twisted to maintain a thread pool. It runs >> database transactions in some available thread via Twisted's >> deferToThread() function. See the transact function in >> http://foss.eepatents.com/sAsync/browser/trunk/sasync/database.py for >> details. >> >> That arrangement created no problem with SQLite and SA 0.1, even though >> SQLite does not allow sharing of connections between threads. Somehow, >> SA 0.1 made the transaction run in its thread with a thread-local copy >> of the engine and table that was created in the main thread. I never >> figured out how, but it just worked. >> >> It doesn't work in SA 0.2, however. I either get the error when trying >> to use the main-thread table in the threaded transaction, or have no >> table available in the thread because it wasn't created there. Now I'm >> trying to figure out how to put humpty dumpty back together again. Any >> suggestions? >> >> Best regards, Ed ------------------------------------------------------- All the advantages of Linux Managed Hosting--Without the Cost and Risk! Fully trained technicians. The highest number of Red Hat certifications in the hosting industry. Fanatical Support. Click to learn more http://sel.as-us.falkag.net/sel?cmd=lnk&kid=107521&bid=248729&dat=121642 _______________________________________________ Sqlalchemy-users mailing list Sqlalchemy-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
