On Mar 19, 2013, at 5:43 PM, Saul St. John wrote: > On 03/19/2013 12:51 PM, Murphy McCauley wrote: >> On Mar 19, 2013, at 10:35 AM, Saul St. John wrote: >> >>> Regarding pt 1: does running POX under PyPy allow for the program to take >>> simultaneous advantage of all the cores in a multiprocessor machine? IOW, >>> does PyPy not suffer from the "single executing thread" limitation that >>> CPython's GIL imposes? >> >> Unfortunately, no. PyPy still has a global lock. Removing the global lock >> in a language designed with one in mind is pretty hard. BUT, you can pass >> the lock between threads better than CPython 2 does. CPython 3 actually >> does better. > > Oh, that is a bummer. I thought I'd read somewhere that transactional > memory was going to obsolete the the need for a global lock someday.
It may, but this is hard. :) Moreover, as I alluded to... POX is written with a pretty straightforward programming model in which you can mostly be unaware of concurrency issues, but this mostly makes sense because there was already the GIL. You could remove the GIL, but to keep the same programming model, you'd be limited in how fully you could leverage multiple cores anyway. That is, the the most obvious way to leverage multiple cores is by having multiple events allowed to fire simultaneously (a la NOX), but that would mean that you need to understand the concurrency issues involved in every event handler. That said, that's not the ONLY way to leverage multiple cores, and we could probably keep the same programming model while still getting SOME advantage (possibly even by doing something as simple as unpacking/packing OpenFlow messages on different cores since this is relatively expensive). >> POX is actually built with the GIL limitation in mind. Really leveraging >> multiple cores would require a more complex programming model. POX keeps >> the simpler model because the limitation is inescapable anyway. > > Could you expand on that a bit? Here's the background: > > In my application, the first packet of every flow needs to go to the > controller, any switch might see that packet, and there might be a ton > of switches. The controller does a fair amount of work on the packets it > sees, but that work will almost certainly not modify global state, and > there's almost no synchronization required between the threads handling > the switches. My understanding is that the GIL is a significant > scalability limitation in this architecture, since it effectively > serializes the processing of OF messages received from any switch, globally. Indeed, the GIL can be a significant scalability limitation here. > I was thinking about whether I could use multiprocessing to launch a > completely separate process for each switch, and use POX more like an > API for OpenFlow, packet parsing, and sockets, than as a > controller-proper. Would that violate POX's programming model? I wouldn't consider using POX as you describe as modifying POX's programming model at all. It does make the sum total programing model of your system somewhat more complex, of course, but that's another issue. This is a valid design as far as I am concerned. If you pursue it and have any suggestions of changes that could be made to POX to support this model better, I'd be interested in hearing them. -- Murphy
