[Python-Dev] Adding Python-Native Threads
There are some very serious problems with threads today. Events are
often proposed as a solution to these problems, but they have their own
issues. A detailed analysis of the two options and some solutions
(most of which are used in my proposal) can be found in Why Events Are
A Bad Idea [0]. I'm going to skip the event aspects and sum up the
thread problems here:
* Expensive (resident memory, address space, time to create and to
switch between)
* Finely-grained atomicity (python is coarser than C, but still finer
than we need)
* Unpredictable (switching between them is not deterministic)
* Uninterruptible (no way to kill them, instead you must set a flag and
make sure they check it regularly)
* Fail silently (if they die with an exception it gets printed to the
console but the rest of the program is left uninformed)
To resolve these problems I propose adding lightweight cooperative
threads to Python. They can be built around a Frame object, suspending
and resuming like generators do.
That much is quite easy. Avoiding the C stack is a bit harder.
However, it can be done if we introduce a new operator for threaded
calls (which I refer to as non-atomic calls), which I propose be
"func@()". As a bonus this prevents any operation which doesn't use
the non-atomic call syntax from triggering a thread switch accidentally.
After that you need a way of creating new threads. I propose we use a
"sibling func@()" statement (note that the "@()" would be parsed as
part of the statement itself, and not as an expression). It would
create a new thread and run func in it, but more importantly it would
also "prejoin" the new thread to the current function. This means that
for the current function to return to its parent, all of the threads it
created must first return themselves. It also means that an exception
in any of them will be propagated to the parent of the creator, after
interrupting all of its siblings (with an Interrupted exception) so
that it is permitted to return.
Now for an example of how this would work in practice, using an echo
server. Note that I am assuming a "con" module (short for concurrent)
that would contain standard functions to support these python-native
threads.
from con import bootstrap, tcplisten, ipv6addr, ipv4addr
def main():
main_s = tcplisten@(ipv6addr('::1', 6839), ipv4addr('127.0.0.1', 6839))
while 1:
sibling echo@(main_s.accept@())
def echo(s):
try:
try:
while 1:
s.write@(s.read@())
except (EOFError, IOError):
pass
finally:
s.close()
if __name__ == '__main__':
try:
bootstrap(main)
except KeyboardInterrupt:
pass
And here is a diagram of the resulting cactus stack, assuming three
clients, two of which are reading and the other is writing.
bootstrap - main - accept
|
echo - read
|
echo - read
|
echo - write
Some notes:
* An idle@() function would be used for all thread switches. I/O
functions such as read@() and write@() would use it internally, and
idle would internally call a function to do poll() on all file
descriptors.
* idle is the function that will raise Interrupted
* It should be illegal to attempt a non-atomic call once your thread is
in an interrupted state. This ensures you cannot get "interrupted
twice", by having something further up the call stack fail and
interrupt you while you are already handling an interruption.
* Being a cooperative system, it does not allow you to use multiple
CPUs simultaneously. I don't regard this as a significant problem,
and in any case there are better ways use multiple CPUs if you need to.
References:
[0]
http://www.usenix.org/events/hotos03/tech/full_papers/vonbehren/vonbehren_html/index.html
--
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python-Native Threads
On 26-jun-2005, at 11:34, Adam Olsen wrote: > > To resolve these problems I propose adding lightweight cooperative > threads to Python. They can be built around a Frame object, > suspending > and resuming like generators do. > > That much is quite easy. Avoiding the C stack is a bit harder. > However, it can be done if we introduce a new operator for threaded > calls (which I refer to as non-atomic calls), which I propose be > "func@()". As a bonus this prevents any operation which doesn't use > the non-atomic call syntax from triggering a thread switch > accidentally. Have a look at stackless python. http://www.stackless.com/ Ronald ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python-Native Threads
Also look at greenlets, they are in the py lib http://codespeak.net/py Regards, Florian Schulze ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python-Native Threads
On 6/26/05, Ronald Oussoren <[EMAIL PROTECTED]> wrote: > Have a look at stackless python. http://www.stackless.com/ On 6/26/05, Florian Schulze <[EMAIL PROTECTED]> wrote: > Also look at greenlets, they are in the py lib http://codespeak.net/py While internally Stackless and Greenlets may (or may not) share a lot with my proposed python-native threads, they differ greatly in their intent and the resulting interface they expose. Stackless and Greenlets are both tools for building an architecture that uses threads, while my python-native threads IS that resulting architecture. For example, with Greenlets you would use the .switch() method of a specific greenlet instance to switch to it, and with my python-native threads you would use the global idle() function which would decide for itself which thread to switch to. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Subclassing in 'C'
Hello, I'm trying to develop an extension module in C, but am having some difficulties. I thought about posting to the general python mailling list, but since it has to do with the C API I figured this was the correct location. My difficulty lies with subclassing a class provided by another external module. I have access to the structure definition, so I can determine the size fo the tp_basicsize member of the PyTypeObject structure. What I can't seem to determine is value for the tp_base element. This is my current "best guess". PyObject* superclass; superclass = PyObject_GetAttrString( module, "SuperClass" ); if ( !superclass ) return; type.tp_base = superclass->ob_type; Py_DECREF( superclass ); type.tp_new = PyType_GenericNew; if ( PyType_Ready( &type ) < 0 ) return; Unfortunately, I seem to be inheriting from the type object for SuperClass, and not from SuperClass itself. I would greatly appreciate some direction on this issue. Thank-you, Robert Johnstone Simon Fraser University http://www.sfu.ca/~rjohnsto/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Subclassing in 'C'
On 23-jun-2005, at 21:05, Robert W. Johnstone wrote: > Hello, > > I'm trying to develop an extension module in C, but am having some > difficulties. I thought about posting to the general python mailling > list, but since it has to do with the C API I figured this was the > correct > location. The python list would have been the right location. > > My difficulty lies with subclassing a class provided by another > external > module. I have access to the structure definition, so I can > determine the > size fo the tp_basicsize member of the PyTypeObject structure. What I > can't seem to determine is value for the tp_base element. > > This is my current "best guess". > > PyObject* superclass; > superclass = PyObject_GetAttrString( module, "SuperClass" ); > if ( !superclass ) return; > type.tp_base = superclass->ob_type; type.tp_base = (PyTypeObject*)superclass; > Py_DECREF( superclass ); And don't decref the superclass. You may want to add a typecheck that checks if the superclass is a type. Ronald ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is PEP 237 final -- Unifying Long Integers and Integers
[me] > > (c) The right place to do the overflow checks is in the API wrappers, > > not in the integer types. [Keith Dart] > That would be the "traditional" method. > > I was trying to keep it an object-oriented API. What should "know" the > overflow condition is the type object itself. It raises OverFlowError any > time this occurs, for any operation, implicitly. I prefer to catch errors > earlier, rather than later. Isn't clear to me at all. I might compute a value using some formula that exceeds 2**32 in some intermediate result but produces an in-range value in the end. That should be acceptable as an argument. I also don't see why one approach is more OO than another; sounds like you have a case of buzzworditis. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python-Native Threads
At 05:04 AM 6/26/2005 -0600, Adam Olsen wrote: >On 6/26/05, Ronald Oussoren <[EMAIL PROTECTED]> wrote: > > Have a look at stackless python. http://www.stackless.com/ > >On 6/26/05, Florian Schulze <[EMAIL PROTECTED]> wrote: > > Also look at greenlets, they are in the py lib http://codespeak.net/py > >While internally Stackless and Greenlets may (or may not) share a lot >with my proposed python-native threads, they differ greatly in their >intent and the resulting interface they expose. Stackless and >Greenlets are both tools for building an architecture that uses >threads, while my python-native threads IS that resulting >architecture. > >For example, with Greenlets you would use the .switch() method of a >specific greenlet instance to switch to it, and with my python-native >threads you would use the global idle() function which would decide >for itself which thread to switch to. See PEP 342 for a competing proposal, that includes a short (<50 lines) co-operative threading example. It uses the 'yield' keyword instead of your '@' proposal, and already has a Sourceforge patch implementing it. It does not propose a specific built-in idler or I/O handler, but such things could be added later. In the short term, I think it's enough that Python be *able* to have such things (without using Stackless or Greenlets), and maybe include an example of a run loop. I don't currently plan to push for a "standard" or "official" run loop implementation in Python 2.5, however, unless all the relevant stakeholders just happen to come to some kind of agreement that makes it happen. Note that while having a universal run loop is a worthy goal, actually implementing one is hard. The experts in this are definitely in the Twisted camp, as they have implemented run loops for numerous platforms to support compatibility w/various GUI frameworks' event loops. To some extent, this means that the run loop needs to be an interface with multiple implementations, and I think that we would need a standard library implementation of PEP 246 in order to support it properly. Anyway, my point is that even deciding on standards for a universal run loop is a fairly big project, one that possibly even deserves its own SIG to put together. By contrast, the implementation of simulated threads and non-blocking function calls was a weekend project. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Some RFE for review
Hi,
while bugs and patches are sometimes tricky to close, RFE can be very easy
to decide whether to implement in the first place. So what about working a
bit on this front? Here are several RFE reviewed, perhaps some can be
closed ("should" is always from submitter's point of view):
1193128:
str.translate(None, delchars) should be allowed and only delete delchars
from the string.
Review: may be a good feature, although string._idmap can be passed as the
first parameter too.
1226256:
The "path" module by Jason Orendorff should be in the standard library.
http://www.jorendorff.com/articles/python/path/
Review: the module is great and seems to have a large user base. On c.l.py
there are frequent praises about it.
1216944:
urllib(2) should gain a dict mapping HTTP status codes to the correspondig
status/error text.
Review: I can't see anything speaking against it.
1214675:
warnings should get a removefilter() method. An alternative would be to
fully document the "filters" attribute to allow direct tinkering with it.
Review: As mwh said in a comment, this isn't Java, so the latter may be
the way to go.
1205239:
Shift operands should be allowed to be negative integers, so e.g.
a << -2 is the same as a >> 2.
Review: Allowing this would open a source of bugs previously well identifiable.
1152248:
In order to read "records" separated by something other than newline, file
objects
should either support an additional parameter (the separator) to (x)readlines(),
or gain an additional method which does this.
Review: The former is a no-go, I think, because what is read won't be lines.
The latter is further complicating the file interface, so I would follow the
principle that not every 3-line function should be builtin.
1110010:
A function "attrmap" should be introduced which is used as follows:
attrmap(x)['att'] == getattr(x, 'att')
The submitter mentions the use case of new-style classes without a __dict__ used
at the right of %-style string interpolation.
Review: I don't know whether this is worth it.
1052098:
An environment variable should be supported to set the default encoding.
Review: If one wants this for a single application, he can still implement it
himself.
985094:
getattr should support a callable as the second argument, used as follows:
getattr(obj, func) == func(obj)
Review: Highly unintuitive to me.
That's all for today; sorry if it was too much ;)
Reinhold
--
Mail address is perfectly valid!
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding the 'path' module (was Re: Some RFE for review)
At 06:57 PM 6/26/2005 +0200, Reinhold Birkenfeld wrote:
>1226256:
>The "path" module by Jason Orendorff should be in the standard library.
>http://www.jorendorff.com/articles/python/path/
>Review: the module is great and seems to have a large user base. On c.l.py
>there are frequent praises about it.
I would note that there are some things in the interface that should be
cleaned up before it becomes a stdlib module. It has many ways to do the
same thing, and many of its property and method names are confusing because
they either do the same thing as a standard function, but have a different
name (like the 'parent' property that is os.path.dirname in disguise), or
they have the same name as a standard function but do something different
(like the 'listdir()' method that returns full paths rather than just
filenames). I'm also not keen on the fact that it makes certain things
properties whose value can change over time; i.e. ctime/mtime/atime and
size really shouldn't be properties, but rather methods. I'm also not sure
how I feel about all the read/write methods that hide the use of file
objects; these seem like they should remain file object methods, especially
since PEP 343 will allow easy closing with something like:
with closing(some_path.open('w')) as f:
f.write(data)
Granted, this is more verbose than:
some_path.write_bytes(data)
But brevity isn't always everything. If these methods are kept I would
suggest using different names, like "set_bytes()", "set_text()", and
"set_lines()", because "write" sounds like something you do on an ongoing
basis to a stream, while these methods just replace the file's entire contents.
Aside from all these concerns, I'm +1 on adding the module.
Here's my list of suggested changes:
* path.joinpath(*args) -> path.subpath(*args)
* path.listdir() -> path.subpaths()
* path.splitall()-> path.parts()
* path.parent-> path.dirname (and drop dirname() method)
* path.name -> path.filename (and drop basename() method)
* path.namebase -> path.filebase (maybe something more descriptive?)
* path.atime/mtime/ctime -> path.atime(), path.mtime(), path.ctime()
* path.size -> path.filesize()
* drop getcwd(); it makes no sense on a path instance
* add a samepath() method that compares the absolute, case and
path-normalized versions of two paths, and a samerealpath() method that
does the same but with symlinks resolved.
And, assuming these file-content methods are kept:
* path.bytes() -> path.get_file_bytes()
* path.write_bytes() -> path.set_file_bytes() and path.append_file_bytes()
* path.text() -> path.get_file_text()
* path.write_text()-> path.set_file_text() and path.append_file_text()
* path.lines() -> path.get_file_lines()
* path.write_lines() -> path.set_file_lines() and path.append_file_lines()
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python-Native Threads
On 6/26/05, Adam Olsen <[EMAIL PROTECTED]> wrote: ... > To resolve these problems I propose adding lightweight cooperative > threads to Python. Speaking of lightweight cooperative threads - has anyone recently tried to build Python with the pth option? It doesn't quite work out of the box. How much maintenance would be required to make it work again? Oren ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding the 'path' module (was Re: Some RFE for review)
Phillip J. Eby wrote: > At 06:57 PM 6/26/2005 +0200, Reinhold Birkenfeld wrote: >>1226256: >>The "path" module by Jason Orendorff should be in the standard library. >>http://www.jorendorff.com/articles/python/path/ >>Review: the module is great and seems to have a large user base. On c.l.py >>there are frequent praises about it. [...] > Aside from all these concerns, I'm +1 on adding the module. > > Here's my list of suggested changes: [...] I agree with your changes list. One more issue is open: the one of naming. As "path" is already the name of a module, what would the new object be called to avoid confusion? pathobj? objpath? Path? Reinhold -- Mail address is perfectly valid! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding the 'path' module (was Re: Some RFE for review)
On Sun, 26 Jun 2005, Phillip J. Eby wrote: > * drop getcwd(); it makes no sense on a path instance Personally I use path.getcwd() as a class method all the time. It makes as much sense as fromkeys() does on a dict instance, which is technically possible but non-sensical. > And, assuming these file-content methods are kept: > > * path.bytes() -> path.get_file_bytes() > * path.write_bytes() -> path.set_file_bytes() and path.append_file_bytes() > * path.text() -> path.get_file_text() > * path.write_text()-> path.set_file_text() and path.append_file_text() > * path.lines() -> path.get_file_lines() > * path.write_lines() -> path.set_file_lines() and path.append_file_lines() I don't know how often these are used. I don't use them myself. I am mainly interested in this module so that I don't have to use os.path anymore. Reinhold Birkenfeld wrote: > One more issue is open: the one of naming. As "path" is already the > name of a module, what would the new object be called to avoid > confusion? pathobj? objpath? Path? I would argue for Path. It fits with the recent cases of: from sets import Set from decimal import Decimal -- Michael Hoffman <[EMAIL PROTECTED]> European Bioinformatics Institute ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding the 'path' module (was Re: Some RFE for review)
Phillip> It has many ways to do the same thing, and many of its property Phillip> and method names are confusing because they either do the same Phillip> thing as a standard function, but have a different name (like Phillip> the 'parent' property that is os.path.dirname in disguise), or Phillip> they have the same name as a standard function but do something Phillip> different (like the 'listdir()' method that returns full paths Phillip> rather than just filenames). To the extent that the path module tries to provide a uniform abstraction that's not saddled with a particular way of doing things (e.g., the Unix way or the Windows way), I don't think this is necessarily a bad thing. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] python/dist/src/Tools/bgen/bgen bgenGenerator.py, 1.17, 1.18 bgenObjectDefinition.py, 1.29, 1.30 bgenType.py, 1.15, 1.16 bgenVariable.py, 1.6, 1.7 scantools.py, 1.37
On 24-jun-2005, at 21:46, [EMAIL PROTECTED] wrote: > Update of /cvsroot/python/python/dist/src/Tools/bgen/bgen > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv18095/Tools/ > bgen/bgen > > Modified Files: > bgenGenerator.py bgenObjectDefinition.py bgenType.py > bgenVariable.py scantools.py > Log Message: > Normalize whitespace to avoid offending Bug Day volunteers. Argh, sorry... I thought I had all my machines educated to do tab expansion for Python, but apparently not... -- Jack Jansen, <[EMAIL PROTECTED]>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding the 'path' module (was Re: Some RFE for review)
Phillip J. Eby wrote: > [...] > I'm also not keen on the fact that it makes certain things > properties whose value can change over time; i.e. ctime/mtime/atime > and > size really shouldn't be properties, but rather methods. I think ctime, mtime and atime should be (or return) datetime.datetime objects instead of integer timestamps. Bye, Walter Dörwald ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] python/dist/src/Tools/bgen/bgen bgenGenerator.py, 1.17, 1.18 bgenObjectDefinition.py, 1.29, 1.30 bgenType.py, 1.15, 1.16 bgenVariable.py, 1.6, 1.7 scantools.py, 1.37
[EMAIL PROTECTED] >> Update of /cvsroot/python/python/dist/src/Tools/bgen/bgen >> In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv18095/Tools/ >> bgen/bgen >> >> Modified Files: >> bgenGenerator.py bgenObjectDefinition.py bgenType.py >> bgenVariable.py scantools.py >> Log Message: >> Normalize whitespace to avoid offending Bug Day volunteers. [Jack Jansen] > Argh, sorry... I thought I had all my machines educated to do tab > expansion for Python, but apparently not... You probably do. I run Tools/scripts/reindent.py over the whole source tree from time to time, and check in whatever it changes. Converting tabs to 4-space indents is one of the things it does, but not the only thing. Most often these days it finds trailing whitespace at the end of a line, and snips it off. Once in a great while, that catches code that *depends* on trailing whitespace (typically in a long string literal), which is viciously obscure (nobody looking at the file can see the dependence, and editors may remove it). It also removes empty lines at the end of a .py file, ensures that the final line ends with a newline, and changes "oddball" indent levels to 4-space indents (curiously, that one triggers 2nd-most often these days). In any case, it's not like this is work -- no need to feel apologetic. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] python/dist/src/Lib Cookie.py, 1.17, 1.18
[EMAIL PROTECTED] > Update of /cvsroot/python/python/dist/src/Lib > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv4891/Lib > > Modified Files: >Cookie.py > Log Message: > bug [ 1108948 ] Cookie.py produces invalid code > > > > Index: Cookie.py > === > RCS file: /cvsroot/python/python/dist/src/Lib/Cookie.py,v > retrieving revision 1.17 > retrieving revision 1.18 > diff -u -d -r1.17 -r1.18 > --- Cookie.py 20 Oct 2003 14:01:49 - 1.17 > +++ Cookie.py 26 Jun 2005 21:02:49 - 1.18 > @@ -470,9 +470,9 @@ > def js_output(self, attrs=None): > # Print javascript > return """ > - > +