Hi guys!
(I'm CCing Thomas, Dan, and Jakob since this mail is partly for them.)
At a SIMAP meeting yesterday, it was discussed if Python was a
requirement for Twisted of if it could be done in any language (Java).
I guess I might have presented Python+Twisted as a kind of deep magic
which solves all problems with asynchronous programming :-)
The use of Deferreds in Twisted sure feels like magic at first, but
let me show you a simplified version of Deferred and DeferredList
written from scratch in 35 lines of Python:
class Share:
def __init__(self):
self.callbacks = []
self.value = None
def addCallback(self, func, *extra_args):
self.callbacks.append((func, extra_args))
def callback(self, initial_value):
self.value = initial_value
for func, extra_args in self.callbacks:
self.value = func(self.value, *extra_args)
def __str__(self):
if self.value is None:
return "Share (no value yet)"
else:
return "Share (value: %s)" % (self.value,)
class ShareList(Share):
def __init__(self, shares):
Share.__init__(self)
self.results = [None] * len(shares)
self.missing_results = len(shares)
for index, share in enumerate(shares):
share.addCallback(self._callback_fired, index)
def _callback_fired(self, result, index):
self.results[index] = result
self.missing_results -= 1
if self.missing_results == 0:
self.callback(self.results)
return result
# Simple addition and multiplication of Share objects:
def add(x, y):
sum = ShareList([x, y])
sum.addCallback(lambda results: results[0] + results[1])
return sum
def mul(x, y):
product = ShareList([x, y])
product.addCallback(lambda results: results[0] * results[1])
return product
#### Using the Shares:
a = Share()
b = Share()
c = Share()
print "Empty shares:"
print "a:", a
print "b:", b
print "c:", c
d = add(a, b)
e = mul(d, c)
print "Scheduled add and mul:"
print "a:", a
print "b:", b
print "c:", c
print "d:", d
print "e:", e
a.callback(10)
b.callback(15)
print "Input 10 into a, 15 into b:"
print "a:", a
print "b:", b
print "c:", c
print "d:", d
print "e:", e
c.callback(20)
print "Input 20 into c:"
print "a:", a
print "b:", b
print "c:", c
print "d:", d
print "e:", e
You can follow along by loading the file in an interactive Python
session. Simply run 'python -i mini-twisted.py' in your terminal.
I'll describe the code below, first the Share class:
class Share:
def __init__(self):
self.callbacks = []
self.value = None
def addCallback(self, func, *extra_args):
self.callbacks.append((func, extra_args))
def callback(self, initial_value):
self.value = initial_value
for func, extra_args in self.callbacks:
self.value = func(self.value, *extra_args)
return self.value
def __str__(self):
if self.value is None:
return "Share (no value yet)"
else:
return "Share (value: %s)" % (self.value,)
This defines a minimal implementation of what Twisted calls a
Deferred. A Share object is a guy which holds a value (initialized to
None in the __init__ constructor) and a list of callbacks (initialized
to the empty list).
You give a Share its value by calling its callback method. This
triggers the evaluation of the callbacks attached to the Share. The
first callback is called with the initial value given, the second
callback is called with the return value of the first, and so on.
This simple mechanism allows you to do boring stuff like this:
>>> x = Share()
>>> print x
Share (no value yet)
>>> x.addCallback(lambda value: value**3)
>>> x.callback(-3)
>>> print x
Share (value: -27)
Nothing really fancy going on here. The fun starts when we can combine
several Shares into a new Share which trigger when the other Shares
gets their value. This ShareList class does that:
class ShareList(Share):
def __init__(self, shares):
Share.__init__(self)
self.results = [None] * len(shares)
self.missing_results = len(shares)
for index, share in enumerate(shares):
share.addCallback(self._callback_fired, index)
def _callback_fired(self, result, index):
self.results[index] = result
self.missing_results -= 1
if self.missing_results == 0:
self.callback(self.results)
return result
The class takes a list of Shares in the constructor and allocates a
list of the same length to hold the results. It then adds a callback
to _callback_fired to each share. This method stores the result in the
correct position and decrements the number of missing shares. If no
shares are missing, we call callback, passing the list of results.
Testing this could look like this:
>>> x = Share()
>>> y = Share()
>>> z = Share()
>>> share_list = ShareList([x, y, z])
>>> share_list.addCallback(lambda results: "Results: %s" % results)
>>> print share_list
Share (no value yet)
>>> x.callback('xxx')
>>> y.callback('yyy')
>>> z.callback(123)
>>> print share_list
Share (value: Results: ['xxx', 'yyy', 123])
The ShareList machinery allows us to define addition and
multiplication of Shares like this:
def add(x, y):
sum = ShareList([x, y])
sum.addCallback(lambda results: results[0] + results[1])
return sum
def mul(x, y):
product = ShareList([x, y])
product.addCallback(lambda results: results[0] * results[1])
return product
And so we can finally do stuff like this:
>>> a = Share()
>>> b = Share()
>>> c = Share()
>>>
>>> d = add(a, b)
>>> e = mul(d, c)
>>>
>>> a.callback(10)
>>> b.callback(15)
>>> c.callback(20)
>>>
>>> print a
Share (value: 10)
>>> print b
Share (value: 15)
>>> print c
Share (value: 20)
>>> print d
Share (value: 25)
>>> print e
Share (value: 500)
To connect this to network programming, simply imagine that there is a
big event loop outside the program which calls callback on the shares
at the right moments with the right values.
That is basically how VIFF works. There is of course more stuff in the
Twisted Deferred than presented here -- error handling for example --
but this is the most important part.
--
Martin Geisler
_______________________________________________
viff-devel mailing list (http://viff.dk/)
[email protected]
http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
