>From my understanding, this should be safe with Twisted:
- SqlAlchemy "engine" operations, where you don't use the ORM and
encapsulate every ansync "chunk" in a transaction, and within a thread
- SqlAlchemy "orm" operations, where every async "chunk" is encapsulated in
it's own session, and within a thread.
I'm not sure how familiar you are with twisted, so i'm going to go real
basic here...
In the way you'd likely use it, Twisted works on the concept of async
'deferreds'. You sort of play "Choose your own adventure" and add in
Callbacks or Errbacks to your miscellaneous functions.
Processing a request wouldn't be:
def process_request():
hit_api
<blocked>
process_results
it would be more like (and I'm butchering this syntax):
def process_request():
d = deferred( get_api ).addCallback(got_api)
d.deferToThread()
def get_api()
<block>
def got_api():
process data
You'd also probably use a DeferredList, which lets you queue up 2dozen
items in a batch at once. it's rad.
Anyways, you wouldn't want to use a SqlAlchemy session that persists
throughout the callback chain. Because everything is so anync , you're
virtually guaranteed that the Session will not match the datastore.
SqlAlchemy also has a blocking interface to the datastore, so IIRC, you not
have issues with all the sessions, but you end up blocking the main twisted
thread.
so the popular workaround(s) are to do both:
1- defer to thread (which most people would do anyways)
2- open/close multiple sessions (which are cheap). basically , instead of
approaching a "task" in twisted as a single action of multiple components,
think of it like a website , with each action as a new page request. your
"web session" is the same, but your "SqlAlchemy Session" changes on each
request.
Anyways, I would probably look more at Celery for what you're looking to
accomplish. Celery is super simple to learn and maintain; On a
learning-curve scale of 1-10, I'd probably put celery at 1, gevent at 5,
and twisted at 10.
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.
