Hello Everyone! My name is Dirk Moors, and since 4 years now, I've been involved in developing a cloud computing platform, using Python as the programming language. A year ago I discovered Twisted Python, and it got me very interested, upto the point where I made the decision to convert our platform (in progress) to a Twisted platform. One year later I'm still very enthousiastic about the overal performance and stability, but last week I encountered something I did't expect;
It appeared that it was less efficient to run small "atomic" operations in
different deferred-callbacks, when compared to running these "atomic"
operations together in "blocking" mode. Am I doing something wrong here?
To prove the problem to myself, I created the following example (Full
source- and test code is attached):
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
import struct
def int2binAsync(anInteger):
def packStruct(i):
#Packs an integer, result is 4 bytes
return struct.pack("i", i)
d = defer.Deferred()
d.addCallback(packStruct)
reactor.callLater(0,
d.callback,
anInteger)
return d
def bin2intAsync(aBin):
def unpackStruct(p):
#Unpacks a bytestring into an integer
return struct.unpack("i", p)[0]
d = defer.Deferred()
d.addCallback(unpackStruct)
reactor.callLater(0,
d.callback,
aBin)
return d
def int2binSync(anInteger):
#Packs an integer, result is 4 bytes
return struct.pack("i", anInteger)
def bin2intSync(aBin):
#Unpacks a bytestring into an integer
return struct.unpack("i", aBin)[0]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
While running the testcode I got the following results:
(1 run = converting an integer to a byte string, converting that byte string
back to an integer, and finally checking whether that last integer is the
same as the input integer.)
*** Starting Synchronous Benchmarks. *(No Twisted => "blocking" code)*
-> Synchronous Benchmark (1 runs) Completed in 0.0 seconds.
-> Synchronous Benchmark (10 runs) Completed in 0.0 seconds.
-> Synchronous Benchmark (100 runs) Completed in 0.0 seconds.
-> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159
seconds.
-> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722
seconds.
-> Synchronous Benchmark (100000 runs) Completed in 0.362999916077
seconds.
*** Synchronous Benchmarks Completed in* 0.406000137329* seconds.
*** Starting Asynchronous Benchmarks . *(Twisted => "non-blocking" code)*
-> Asynchronous Benchmark (1 runs) Completed in 34.5090000629 seconds.
-> Asynchronous Benchmark (10 runs) Completed in 34.5099999905 seconds.
-> Asynchronous Benchmark (100 runs) Completed in 34.5130000114 seconds.
-> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657 seconds.
-> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924 seconds.
-> Asynchronous Benchmark (100000 runs) Completed in 41.492000103 seconds.
*** Asynchronous Benchmarks Completed in *42.1460001469* seconds.
Am I really seeing factor 100x??
I really hope that I made a huge reasoning error here but I just can't find
it. If my results are correct then I really need to go and check my entire
cloud platform for the places where I decided to split functions into atomic
operations while thinking that it would actually improve the performance
while on the contrary it did the opposit.
I personaly suspect that I lose my cpu-cycles to the reactor scheduling the
deferred-callbacks. Would that assumption make any sense?
The part where I need these conversion functions is in marshalling/protocol
reading and writing throughout the cloud platform, which implies that these
functions will be called constantly so I need them to be superfast. I always
though I had to split the entire marshalling process into small atomic
(deferred-callback) functions to be efficient, but these figures tell me
otherwise.
I really hope someone can help me out here.
Thanks in advance,
Best regards,
Dirk Moors
twistedbenchmark.py
Description: Binary data
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
