I've spent today repeating Eric Day's performance benchmarks, including simple python threading, on a quad core instead of a dual core (I actually wanted to evaluate Jython, but I digress.) Turns out threading then beats Twisted and Eventlet. Twisted can get the upper hand again over threading, with a multi-process model, but we're not talking significant differences (64 seconds vs 52 seconds).
My conclusion is simple: it isn't clear that Twisted buys us the great performance it promises. It is clear that it locks us in to the Twisted deferred model. If we stick with 'clean code', we bet on 3 horses: - Python 3 may fix some of the GIL issues that make threaded code perform poorly, in particular the ones that David Beazley recently highlighted (though this will depend on how much time the apps spends on CPU vs I/O) - Jython doesn't even have a GIL - Eventlet can offer async without code changes If we choose Twisted, we're betting everything on Twisted being the best option. If the Twisted proponents want to demonstrate a benchmark that simulates the nova-api use case, where Twisted offers a sufficient performance benefit (over both eventlet and simple threading) to justify the lock-in and the code complexity, _then_ we should consider Twisted. Until that result happens, I vote that we should avoid the Twisted lock-in. To me, the language choice of Python indicates that we care about developer productivity and code simplicity & correctness more than we care about raw performance. IMHO, choosing Twisted is inconsistent with that choice. Incidentally, this talk is a real eye opener: http://blip.tv/file/2232349 It's the 'opening act' to David Beazley's GIL talk, and describes very well how Python threading is actually much better than generally thought. Justin On Sat, Aug 7, 2010 at 6:03 AM, Rick Clark <[email protected]> wrote: > On 08/04/2010 01:35 PM, Gustavo Niemeyer wrote: > > I'm not actually arguing either way, but if I had to make this > > decision, I'd have a slightly more even comparison between eventlet > > and twisted. Twisted is stable, well known, has several maintainers, > > and *many* users. Canonical, Google, Apple, ITA, and many many other > > use Twisted. You can trust on it being around and well maintained in > > the foreseeable future. > > I am not making a judgment here, but I agree with Gustavo. I think the > health and viability of a project should be considered in it's > selection. The decision isn't purely technical. > > On another note, this is not an either/or question. Twisted does many > things that eventlet does not, they are not mutually exclusive. > > > _______________________________________________ > Mailing list: https://launchpad.net/~nova > Post to : [email protected] > Unsubscribe : https://launchpad.net/~nova > More help : https://help.launchpad.net/ListHelp > >
_______________________________________________ Mailing list: https://launchpad.net/~nova Post to : [email protected] Unsubscribe : https://launchpad.net/~nova More help : https://help.launchpad.net/ListHelp
