Don Dwiggins <[email protected]> writes:
> Hmmm, Could you elaborate a bit about the "self-contained"? I do have
> some cases where I fire off a deferredList with several (independent)
> queries in it. Might that be problematic?
My guess is that your "independent" comment is essentially the same as
my "self-contained".
The point I was making was that each time you delegate a request to
the database thread, it queues up with other requests (which may be
from different deferred chains) and is then executed FIFO from the
queue. In this context, a "request" is the callable that will be
executed within the database thread context, so it can can still have
multiple individual SA database operations, all as part of the same
request.
>From the SA perspective, everything executes in the single background
thread via the same database connection and engine. So you don't want
two separate requests to that thread to have any coupling - the
simplest example of which would be them requiring that they both
execute in the same transaction. You can't guarantee the integrity of
that, at least with my current implementation, since some other
request might hit the queue in between the original requests, and then
get caught up in the transaction.
So for example, (warning non-tested pseudo code), if you're using a
deferredList to start and retrieve results from functions "a" and "b",
and assuming "db" is the SA database object, the following could in
theory be risky, if the a_* or b_* callbacks were coupled in some way.
def mainfunc():
d = deferredList([a(), b()])
return d
def a():
d = db.run(a_first_func)
d.addCallback(a_process_first_result)
d.addCallback(db.run, a_second_func)
d.addCallback(a_process_second_result)
return d
def b():
d = db.run(b_first_func)
d.addCallback(b_process_first_result)
d.addCallback(db.run, b_second_func)
d.addCallback(b_process_second_result)
return d
That's because at the database layer, this could (and probably would
in this example) execute as:
a_first_func
b_first_func
a_second_func
b_second_func
One way to restructure this, depending on what the functions involved
do, would be (I've kept the function calls separate just for clarity):
def mainfunc():
d = deferredList([a(), b()])
return d
def _a_dbop():
r1 = a_first_func()
r2 = a_process_first_result(r1)
r3 = a_second_func(r2)
return a_process_second_result(r3)
def a():
d = db.run(_a_dbop)
return d
def _b_dbop():
r1 = b_first_func()
r2 = b_process_first_result(r1)
r3 = b_second_func(r2)
return b_process_second_result(r3)
def b():
d = db.run(_b_dbop)
return d
Now the first_func/second_func's in each case always run together
within a single database thread request. In some cases this can even
simplify things since what used to be a deferred callback chain
becomes inline code since you're already in a separate thread
(eseentially like a deferToThread). Of course, the above may not be
practical if the *_process_*_result functions need access to your
Twisted code, in which case you're just back to being careful not to
require a particular execution order among the various database
callables.
In practice I haven't found this to be a particularly onerous
restriction in my own code, especially when sticking to the SA SQL
layer.
Basically the granularity of the sharing of the database thread (and
SA's connection to the database) is that of the callable you pass to
the database thread to execute. As long as you implement those
callables so they can run independently, you should be fine.
Make sense?
-- David
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~----------~----~----~----~------~----~------~--~---
