Re: [Stackless] Multi-CPU Actor Based Python

2008-11-21 Thread OvermindDL1
I do not know if you all are still dealing with this, but I made a library on top of Stackless Python that I named Pylang. It is basically a more 'Erlang'ish interface for Stackless, basically instead of the channel model that Stackless uses, it exposed an Actor model, worked near identically to E

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-20 Thread Timothy Baldridge
> Where does this 90% number come from? I was under the impression that > if the GIL were removed -- the vast majority of lock activity by finer > grained locking would be around the object reference counters. How > does multi-threaded actor-based concurrency help with that? The 90% come right out

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-20 Thread Arnar Birgisson
Hi all, Haven't read all of this thread carefully, but.. On Tue, Nov 18, 2008 at 17:32, Timothy Baldridge <[EMAIL PROTECTED]> wrote: > Because all information for an object is contained in that object > (including message queues, and data). We could remove all locks from > the list objects, as th

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-20 Thread niki
Timothy Baldridge wrote: POSH is more based around the idea of putting some objects in OS shared memory. This would boost performance in some cases, as objects no longer need to be serialized when data is changed. Depending on the speed of the (de)allocation, it might actually work, I'm going to

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Jeff Senn
On Nov 19, 2008, at 1:01 PM, Larry Dickson wrote: Is there a non-object-oriented flavor of Stackless Python? I've run into this sort of thing before. OO techniques seem to require extreme centralization, which kills "fast", makes "safe" impossible in the real world, and I'm not even clear

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Chris McCormick
On Wed, Nov 19, 2008 at 05:44:46PM +, Michael wrote: > On Wednesday 19 November 2008 15:50:08 Jeff Senn wrote: > ... > > and perhaps you shouldn't be discouraged by no-sayers... you might > > just invent something wonderful... > > +1 Maybe I'm way off the mark, and this is obvious to everyone

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Larry Dickson
Is there a non-object-oriented flavor of Stackless Python? I've run into this sort of thing before. OO techniques seem to require extreme centralization, which kills "fast", makes "safe" impossible in the real world, and I'm not even clear on "nice-looking"... On 11/19/08, Jeff Senn <[EMAIL PROTEC

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Gabriel Lavoie
Hello everyone, I really find this discussion interesting because it's somewhat related to my university master degree project at Université Laval, Québec, QC, Canada. Right now, I'm working on a simple way to extend channel communication between different Python VM. My work probably isn't rea

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Michael
On Wednesday 19 November 2008 15:50:08 Jeff Senn wrote: ... > and perhaps you shouldn't be discouraged by no-sayers... you might > just invent something wonderful... +1 Michael -- http://yeoldeclue.com/blog http://www.kamaelia.org/Home ___ Stackless

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Timothy Baldridge
> BTW: I'd love to have a Stackless Python w/o a GIL... I just > can't afford to do the (ton of) work! I gather the "posh" thing > Michael mentioned is a hack to put separate locks (LILs? :-) ) > around pieces of memory that contain python objects -- seems like > a lot of hoops to jump through, fo

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Jeff Senn
On Nov 19, 2008, at 9:49 AM, Timothy Baldridge wrote: What I'm more looking to reduce, is the overhead of transferring data from one Python VM to another. In current implementations transferring data from one VM to another requires pickling data (and that requires traversing the entire object b

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Larry Dickson
The key word ought to be "stackless" as in "Stackless Python." The Transputer and occam people showed long ago that multiprocessing (which is what you are really talking about here) as well as multitasking become really easy (i.e. overhead of a dozen cycles or so) if you get rid of implicitly share

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Timothy Baldridge
What I'm more looking to reduce, is the overhead of transferring data from one Python VM to another. In current implementations transferring data from one VM to another requires pickling data (and that requires traversing the entire object being transmitted, pickling each part along the way), trans

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-19 Thread Michael
On Tuesday 18 November 2008 21:41:36 Timothy Baldridge wrote: > The issue with Stackless is that you are still limited by the GIL. > Therefore you can never use more than 100% of one CPU. 1 hardware thread can only give you 100% of itself. (I'm using that rather than saying core or CPU since it'

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Emil Friðriksson
I'd like to point out that Richard is a programmer at CCP and worked on game logic among other things. I'm also an employee of CCP and one of the people that run the live game cluster. We run 1 process, running one stackless python thread per CPU and to that instance of the interpreter can we ma

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Timothy Baldridge
> There is not a one to one mapping between threads and solar systems. I beg to differ. Unless something has changed: "Each of EVE's 5000+ star systems is loaded as a separate process onto any one of hundreds of IBM blade servers, with some high-load systems being given a server all to themselves

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Richard Tew
On Tue, Nov 18, 2008 at 4:41 PM, Timothy Baldridge <[EMAIL PROTECTED]> wrote: > But that's the issue, even CCP has had issues scaling EVE. EVE is > implemented in a way that each star system is handled by a single > thread. This is fine if you have 1 star systems and 100 CPUs. But > the problem

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Michael
On Tuesday 18 November 2008 20:33:24 Timothy Baldridge wrote: > but passing messages in between > objects. If we can get a framework to match this model, I think it > could be quite usable. Stackless essentially does that in the form of tasklets using channels. Kamaelia also does something simila

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Timothy Baldridge
> Stackless essentially does that in the form of tasklets using channels. > Kamaelia also does something similar, but uses lists for generators, > Queue.Queue's for threads, the os.pipes for (pickled) objects between > processes. (kamaelia works with standard cpython) The issue with Stackless is t

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Timothy Baldridge
> Hm. Are you forgetting that "objects" in Python are all by reference. > And therefore it is pretty much impossible to "contain" them. > > Imagine I create a message: > > msg = [ [1,2], [3,4] ] # Notice msg is a reference to a tree of 7 objects > That's a good point, and one I hadn't really th

Re: [Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Jeff Senn
On Nov 18, 2008, at 11:32 AM, Timothy Baldridge wrote: Because all information for an object is contained in that object (including message queues, and data). We could remove all locks from the list objects, as the internal object tasklet will be the only object modifying the code. These queues

[Stackless] Multi-CPU Actor Based Python

2008-11-18 Thread Timothy Baldridge
I've been working on getting a actor based framework setup and working in Stackless Python. A true object-oriented, actor based framework I would allow us to get rid of the GIL in python (for the most part) but still keep the routines just as fast as they are now. For information on Actor based mod