Hi Trent,
I just started to try to understand the idea and the implications.
Removing almost all of your message since that is already too long
to work with:
The reference is
http://mail.python.org/pipermail/python-dev/2013-March/124690.html
On 3/14/13 11:45 AM, Trent Nelson wrote:
On Wed, Mar 13, 2013 at 07:05:41PM -0700, Trent Nelson wrote:
Just posted the slides for those that didn't have the benefit of
attending the language summit today:
https://speakerdeck.com/trent/parallelizing-the-python-interpreter-an-alternate-approach-to-async
Someone on /r/python asked if I could elaborate on the "do Y" part
of "if we're in a parallel thread, do Y, if not, do X", which I
(inadvertently) ended up replying to in detail. I've included the
response below. (I'll work on converting this into a TL;DR set of
slides soon.)
Can you go into a bit of depth about "X" here?
That's a huge topic that I'm hoping to tackle ASAP. The basic premise
is that parallel 'Context' objects (well, structs) are allocated for
each parallel thread callback. The context persists for the lifetime of
the "parallel work".
<big snip>
So, the remaining challenge is preventing the use case alluded to
earlier where someone tries to modify an object that hasn't been "async
protected". That's a bit harder. The idea I've got in mind is to
instrument the main CPython ceval loop, such that we do these checks as
part of opcode processing. That allows us to keep all the logic in the
one spot and not have to go hacking the internals of every single
object's C backend to ensure correctness.
Now, that'll probably work to an extent. I mean, after all, there are
opcodes for all the things we'd be interested in instrumenting,
LOAD_GLOBAL, STORE_GLOBAL, SETITEM etc. What becomes challenging is
detecting arbitrary mutations via object calls, i.e. how do we know,
during the ceval loop, that foo.append(x) needs to be treated specially
if foo is a main-thread object and x is a parallel thread object?
There may be no way to handle that *other* than hacking the internals of
each object, unfortunately. So, the viability of this whole approach
may rest on whether or that's deemed as an acceptable tradeoff (a
necessary evil, even) to the Python developer community.
This is pretty much my concern:
In order to make this waterproof, as required for CPython, you will quite
likely have to do something on very many objects, and this is hard
to chime into CPython.
If it's not, then it's unlikely this approach will ever see the light of
day in CPython. If that turns out to be the case, then I see this
project taking the path that Stackless took (forking off and becoming a
separate interpreter).
We had that discussion quite often for Stackless, and I would love to find
a solution that allows to add special versions and use cases to CPython
in a way that avoids the forking as we did it.
It would be a nice thing if we could come up with a way to keep CPython
in place, but to swap the interpreter out and replace it with a specialized
version, if the application needs it. I wonder to what extent that would be
possible.
What I would like to achieve, after having given up on Stackless integration
is a way to let it piggyback onto CPython that works like an extension
module, although it hat effectively replace larger parts of the interpreter.
I wonder if that might be the superior way to have more flexibility,
without forcing
everything and all go into CPython.
If we can make the interpreter somehow pluggable at runtime, a lot of issues
would become much simpler.
There's nothing wrong with that; I am really excited about the
possibilities afforded by this approach, and I'm sure it will pique the
interest of commercial entities out there that have problems perfectly
suited to where this pattern excels (shared-nothing, highly concurrent),
much like the relationship that developed between Stackless and Eve
Online.
What do you think: does it make sense to think of a framework that
allows to replace the interpreter at runtime, without making normal
CPython really slower?
cheers - chris
--
Christian Tismer :^) <mailto:tis...@stackless.com>
Software Consulting : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/
14482 Potsdam : PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776 fax +49 (30) 700143-0023
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
