a) they block the current thread of execution (which Twisted cannot deal with).
b) they presumably go to a multithreaded / multiprocess database server, which can run multiple queries simultaneously.
So while you're waiting for your 10-second giant query to finish, you can go do something else, including running another query that will start and finish before the first is done. At any rate, threaded vs. some other async mechanism is kind of a moot point as I think background queries *must* run in a thread in SA to take advantage of the thread-local transaction context that you're going to need for multiple in-flight queries.
I don't know how the idea that SA needs to be done in a threaded "style" got started, but that's not what I'm proposing. All I'd like to do is formalize the concepts of a background query, of somehow marshaling the parameters to that query, and the specification of a callback to receive the query results, so that integrators have an obvious place to hook in.
Rick
On 2/15/06, Valentino Volonghi aka Dialtone <[EMAIL PROTECTED]> wrote:
On Wed, 15 Feb 2006 13:15:58 -0500, Kevin Dangoor <[EMAIL PROTECTED]> wrote:
>Particularly for something like a database layer that runs blocking
>operations (reads from the database) all the time, Twisted does
>require changes to how things work. Someone made an SQLObject-like OR
>mapper for Twisted and they implemented a nice decorator to make the
>usage somewhat palatable:
> http://usrsrc.org/darcs/tada/doc/examples/taoBasicUsage.py
>
>One function with a bunch of yields is a lot nicer than a ton of
>functions (having to create a callback for *each* database call!).
Notice how my decorator doesn't require anything like that. Not only this but
also twisted.enterprise.adbapi doesn't require something like that.
tada developers decided to do a weird thing IMHO, they run in a separate
thread each operation returning a deferred for each AFAIK. Doing this without
a language that supports deferreds natively means that you are actually
putting a lot of overhead on the developer thus their solution with
generators (which is anyway already possible using standard twisted without
their hack). The hinted and recommended approach in handling database
connections in Twisted is either run the database syncronously (if the
database is inprocess like SQLite and you can handle it) or run the
database queries in a separate thread putting all the transaction in the
thread.
>For my own work, I've generally chosen ease-of-coding over the ability
>to handle thousands of simultaneous requests on a single box. (Some
>have argued that multithreaded programs aren't easy, but I guess
>that's all a question of how one's brain works...)
Not really. All the people that really understand threads are the ones that
most of all are aware of the many problems and complexity that they
introduce. As Bruce Eckel reported in a recent python-dev thread, Sun only
managed to get threads right in the last JVM release. Until the last release,
for example, SWING (for example) was known to deadlock while doing
'weird' things.
I also read some people in the turbogears mailing list (or maybe only in the
cherrypy mailing, I can't remember exactly) who suffer from random crash
problems due to threading and thread safeness under medium-high load.
I don't want to start any 'war' between threading and async merits but saying
that threading is easy is just too bold to not answer in some way.
FWIW I find async code to be many times easier to manage and debug (at least
it is deterministic and reproducible) than threading code.
By the way I don't think this sub-discussion has much to do with the real
topic of the thread.
--
Valentino Volonghi aka Dialtone
Now Running MacOSX 10.4
Blog: http://vvolonghi.blogspot.com
New Pet: http://www.stiq.it
