On Mon, Dec 20, 2010 at 12:18 PM, Andy Robinson <a...@reportlab.com> wrote:
> As an attempt to generate some content and balance out the "jobs"
> discussion....
>
> Why don't a few people here tell us what they got up to this year?
> Neat projects at work, things you learned about Python in 2010, things
> you've been playing with....
>
> I'm having a mad day but will try to post mine tonight or tomorrow...
At TweetDeck we've built our REST API on Twisted over Amazon EC2 and
SimpleDB. We also rolled out a scheduled update (delay
tweeting/updating in time to hit other timezones, etc), over RabbitMQ
and Celery. We also continued extending our website, which is
implemented in Django.
Perhaps most interesting out of this is working both with synchronous
and asynchronous code. We have common libraries implementing clients
for dozens of REST apis that we use both on the Django and Twisted
side. You usually have to choose one path or the other, but this year
I've repurposed this common library to do either, depending on an
instantiation argument.
It turned out to not be so bad - but then why would it be? Most
libraries out there blindly implement a synchronous model without
realizing that splitting the request and response code is actually
quite easy - and doing this makes it also easy to convert this to
asynchronous code.
# Usual synchronous code
def get_foo(arg1, arg2):
args = prepare_request()
result = make_blocking_call() # usually uses something from
httplib or httplib2
do_stuff(result)
do_more_stuff(result)
# async friendly version of the above
def get_foo_nicer(arg1, arg2):
args = prepare_request()
result = make_maybe_blocking_call()
def post_process(result):
do_stuff(result)
do_more_stuff(result)
post_process(result)
The second version is exactly the same as the previous, but now it
becomes much easier to make it async (rather, Twisted aware). The key
player to *making* this async-aware is the Deferred - that is, a
"promise" that a result of the request will eventually appear. Then,
you can make the last line slightly more intelligent by calling
another method, which decides to magically apply the post_process
function in the usual synchronous case, or add it as a callback to the
deferred in the asynchronous case.
I've implemented this pattern into boto in a "async aware" branch and
it seems to work quite nicely, without disrupting much code. It
strikes me a something that could easily be emulated in most other
libraries
Anyway, I thought that was mildly interesting and worth sharing. Merry
Christmas everyone, and Happy New Year!
Reza
--
Reza Lotun
mobile: +44 (0)7521 310 763
email: rlo...@gmail.com
work: r...@tweetdeck.com
twitter: @rlotun
_______________________________________________
python-uk mailing list
python-uk@python.org
http://mail.python.org/mailman/listinfo/python-uk
