At both PyCon 2011 and EuroPython 2010, I was somewhat surprised by the
popularity of greenlet based solutions (eventlet, gevent). This is despite the
fact that a Stackless Python based solution ought to outperform the
aforementioned.
During PyCon 2011, there were two discussions of particular interest. The first
was with Christian Tismer about what makes stackless "stackless" and when does
hard switching occur. This is important since soft switching is ten times
faster than hard switching.
Of interest to me is how many times in the typically networking scenario does
hard switches occur? And maybe alter Stackless to occur these cases.
The second talk was with the Twisted team. The subject of a Stackless Twisted
Reactor was mentioned and I was told, "why bother - eventlets has Twisted
support!" From what I have seen, Twisted support is in the form of blockOns, a
technique used by Christopher Armstrong. I use blockOns.
BlockOn is a very effective technique. The bulk of my talk, "Adventures in
Stackless Python / Twisted Integration" deal with handling what for now are
rather obscure edge cases where dead lock would occur. Since that talk,
I have been interested in making tasklets and callbacks work together more
naturally. In part I believe callbacks become unnecessary.
Now this echos back to a conversation I had at EuroPython 2010. Things boiled
down to "why do you need a scheduler?" Well in the case of both gevent and
twisted, there is a still a "scheduler" that is essentially non pre-emptive and
can only schedule I/O bound "tasks." However this may be good enough for most
people's needs.
The Twisted conversation ended with me sitting with the Twisted team and
learning the basics of writing a custom reactor. Glyph and J.P kindly pointed
out the pit-falls.
The plan:
Using stackless.py as a test bed, I would write a Stackless Reactor. However
instead of using CPython with greenlets. I would use PyPy-c which to the best
of my knowledge is solely soft-switching. I would replace the stackless.run()
with a new stackless.run that would be a modified Twisted reactor. Of course
tasklets would not know the difference.
Amongst the things I would do, is replace the blockOn, with a lighter "channel"
class that could transfer data into the tasklet and schedule it. Also
appropriate Twisted class methods could be transparently called, i.e.
client.getPage().
And to cover edge cases, protocol instances would run in a tasklet (after all I
want to build stuff that I can use for my own work).
I have a few more ideas but this all for now. Comments?
Cheers,
Andrew
_______________________________________________
Stackless mailing list
[email protected]
http://www.stackless.com/mailman/listinfo/stackless
