Hi,
Continuing from my previous post, the context of which was eventlet,
here is the reason why I was asking about it.
I have been planning to rewrite HarvestMan using a non-blocking approach
rather than the current design of achieving concurrency using multiple
threads (which do not scale very well in Python).
I have been evaluating twisted for sometime for doing this, but I have not
been able to wrap my brain around twisted. Also the fact that twisted
requires an explicit event loop call is not very good and makes it difficult
to write an application which has its own separate event loop.
Eventlets look promising because it is implemented using coroutines without
an explicit event loop call, instead the event loop is run transparently when
required. Also due to its design, it can be easily merged with Stackless, which
I think it is already on the way to.
Moreover, the sample code shows a crawler example, so there it is!
I am looking at creating another implementation of HarvestMan based on
eventlets rather than threads. This will run parallel to the regular HarvestMan.
The plan is to write a crawler that scales very well w.r.t the number
of requests
and websites.
Let me know if someone would like to be part of this.
Thanks
--
-Anand
_______________________________________________
BangPypers mailing list
BangPypers@python.org
http://mail.python.org/mailman/listinfo/bangpypers
