Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
Hi Skip, I had been approached to do the exact same thing, are you trying to back port the trunk version (2.6) or py3000? On Thu, Oct 16, 2008 at 12:01 PM, Skip Montanaro <[EMAIL PROTECTED]> wrote: > I'd like to try backporting the multiprocessing module to Python 2.4. My > first > problem appears to be the reliance on a complete(?) rewrite of the buffer > stuff. > > Any clues about transforming this code would be much appreciated. > > (Note: I'm backporting because the Python 2.6 version appears to be much more > robust than the 0.52 third-party release.) > > Thanks, > > Skip > ___ > Python-3000 mailing list > [EMAIL PROTECTED] > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: > http://mail.python.org/mailman/options/python-3000/jnoller%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
Also note, for python 2.4/2.5 you are going to *need* the patch to bug http://bugs.python.org/issue874900 On Thu, Oct 16, 2008 at 12:34 PM, Jesse Noller <[EMAIL PROTECTED]> wrote: > Hi Skip, > > I had been approached to do the exact same thing, are you trying to > back port the trunk version (2.6) or py3000? > > On Thu, Oct 16, 2008 at 12:01 PM, Skip Montanaro > <[EMAIL PROTECTED]> wrote: >> I'd like to try backporting the multiprocessing module to Python 2.4. My >> first >> problem appears to be the reliance on a complete(?) rewrite of the buffer >> stuff. >> >> Any clues about transforming this code would be much appreciated. >> >> (Note: I'm backporting because the Python 2.6 version appears to be much more >> robust than the 0.52 third-party release.) >> >> Thanks, >> >> Skip >> ___ >> Python-3000 mailing list >> [EMAIL PROTECTED] >> http://mail.python.org/mailman/listinfo/python-3000 >> Unsubscribe: >> http://mail.python.org/mailman/options/python-3000/jnoller%40gmail.com >> > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Backporting multiprocessing?
I'd like to try backporting the multiprocessing module to Python 2.4. My first problem appears to be the reliance on a complete(?) rewrite of the buffer stuff. Any clues about transforming this code would be much appreciated. (Note: I'm backporting because the Python 2.6 version appears to be much more robust than the 0.52 third-party release.) Thanks, Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
> I had been approached to do the exact same thing, are you trying to > back port the trunk version (2.6) or py3000? I'm trying to backport from 2.6. It appears that the buffer stuff is completely new though (backported from Python 3.0). S ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documentation idea
Raymond Hettinger wrote: > * It will assist pypy style projects and other python implementations > when they have to build equivalents to CPython. > > * Will eliminate confusion about what functions were exactly intended to > do. > > * Will confer benefits similar to test driven development where the > documentation and pure python version are developed first and doctests > gotten to pass, then the C version is created to match. I haven't seen anyone comment about this assertion of "equivalence". Doesn't it strike you as difficult to maintain *two* versions of every function, and ensure they match *exactly*? The utility to PyPy-style projects is minimized if the two version aren't identical. And while it's possible to say, "the tests say they are equiavelent, so they are;" history is quite clear about people depending on "features" that are untested and were unintended side-effects of the manner in which something was implemented. I think it would be a dilution of developer man-hours to force them to maintain two versions in lock-step, and it significantly adds to the burden of writing and reviewing potential bugfixes. While I applaud the idea of documenting C functions in this manner, let's not confuse documentation with equivalence. If the standard distribution of Python exports the C version, then all bets are off whether the Python version is a drop-in replacement (even if the buildbots regularly test them). I feel so strongly about this that I think that the consideration of adding this should be frame /solely/ as a documentation tool and nothing more. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
Do we want to start a google code project for this given all three of us are interested in this? :) On Thu, Oct 16, 2008 at 3:28 PM, Christian Heimes <[EMAIL PROTECTED]> wrote: > Skip Montanaro wrote: >> >> I'd like to try backporting the multiprocessing module to Python 2.4. My >> first >> problem appears to be the reliance on a complete(?) rewrite of the buffer >> stuff. >> >> Any clues about transforming this code would be much appreciated. >> >> (Note: I'm backporting because the Python 2.6 version appears to be much >> more >> robust than the 0.52 third-party release.) > > Good timing, Skip! I was planing to do a backport to 2.5, too. I've some > experience with both the old and the new buffer protocol. I might be of some > assistance to you. > > I like to make as much code of the trunk version compatible with 2.5 and 2.4 > as possible. Let's see how far we can get with a bunch of macros and > #ifdefs. > > Christian > > ___ > Python-3000 mailing list > [EMAIL PROTECTED] > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: > http://mail.python.org/mailman/options/python-3000/jnoller%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
Jesse Noller wrote: Do we want to start a google code project for this given all three of us are interested in this? :) Do we need (yet) another Google code project? Isn't svn.python.org sufficient for our needs? I'm -0 on a Google code project but I'll give you my gmail account if you insist on one. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documentation idea
On Thu, Oct 16, 2008 at 11:13 AM, Scott Dial <[EMAIL PROTECTED]> wrote: > Raymond Hettinger wrote: >> * It will assist pypy style projects and other python implementations >> when they have to build equivalents to CPython. >> >> * Will eliminate confusion about what functions were exactly intended to >> do. >> >> * Will confer benefits similar to test driven development where the >> documentation and pure python version are developed first and doctests >> gotten to pass, then the C version is created to match. > > I haven't seen anyone comment about this assertion of "equivalence". > Doesn't it strike you as difficult to maintain *two* versions of every > function, and ensure they match *exactly*? More time-consuming than difficult. Raymond is currently talking about things like built-ins and methods on types who do not exactly change very often. > The utility to PyPy-style > projects is minimized if the two version aren't identical. And while > it's possible to say, "the tests say they are equiavelent, so they are;" > history is quite clear about people depending on "features" that are > untested and were unintended side-effects of the manner in which > something was implemented. Right, and when we find out that there is a difference, we typically standardize on a specific version and developers using the bogus semantics switch. > I think it would be a dilution of developer > man-hours to force them to maintain two versions in lock-step, and it > significantly adds to the burden of writing and reviewing potential > bugfixes. > Well, I don't see this applying to every extension module in the stdlib that does not already have a pure Python equivalent. This view also assumes that if this position was taken people will continue to write extension modules when they are not necessarily needed. If this actually makes people to write more pure Python code over extension modules I think that is a plus. And Raymond, more than probably anyone, can address the overhead he has faced in maintaining both the pure Python version of itertools in the docs and the extension module. > While I applaud the idea of documenting C functions in this manner, > let's not confuse documentation with equivalence. If the standard > distribution of Python exports the C version, then all bets are off > whether the Python version is a drop-in replacement (even if the > buildbots regularly test them). Well, considering we have not even gotten far enough to actually do this for the documentation case, I think worrying about equivalence might be jumping the gun slightly as it is more work as you point out, Scott. But one thing about doing this is it might draw in the various alternative VM folks to help maintain the Python code. If Jython, IronPython, and/or PyPy actually use the Python code for themselves then I suspect they would help with maintenance. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
On Thu, Oct 16, 2008 at 3:38 PM, Christian Heimes <[EMAIL PROTECTED]> wrote: > Jesse Noller wrote: >> >> Do we want to start a google code project for this given all three of >> us are interested in this? :) > > Do we need (yet) another Google code project? Isn't svn.python.org > sufficient for our needs? I'm -0 on a Google code project but I'll give you > my gmail account if you insist on one. > > Christian > I've not used svn.python.org for personal side/projects - also, ideally the back port would be stand-alone and package-index installable ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documentation idea
Raymond Hettinger wrote:
* It will assist pypy style projects and other python implementations
when they have to build equivalents to CPython.
* Will eliminate confusion about what functions were exactly intended to
do.
* Will confer benefits similar to test driven development where the
documentation and pure python version are developed first and doctests
gotten to pass, then the C version is created to match.
I haven't seen anyone comment about this assertion of "equivalence".
Doesn't it strike you as difficult to maintain *two* versions of every
function, and ensure they match *exactly*?
Glad you brought this up. My idea is to present rough equivalence
in unoptimized python that is simple and clear. The goal is to provide
better documentation where code is more precise than English prose.
That being said, some subset of the existing tests should be runnable
against the rough equivalent and the python code should incorporate doctests.
Running both sets of test should suffice to maintain the rough equivalence.
The notion of exact equivalence should be left to PyPy folks who can attest
that the code can get convoluted when you try to simulate exactly when
error checking is performed, read-only behavior for attributes, and making
the stacktraces look the same when there are errors. In contrast, my
goal is an approximation that is executable but highly readable and expository.
My thought is to do this only with tools where it really does enhance the
documentation. The exercise is worthwhile in and of itself. For example,
I'm working on a pure python version of str.split() and quickly determined
that the docs are *still* in error even after many revisions over the years
(the whitespace version does not, in fact, start by stripping whitespace
from both ends). Here's what I have so far:
def split(s, sep=None, maxsplit=-1):
"""split(S, [sep [,maxsplit]]) -> list of strings
Return a list of the words in the string S, using sep as the
delimiter string. If maxsplit is given, at most maxsplit
splits are done. If sep is not specified or is None, any
whitespace string is a separator and empty strings are removed
from the result.
>>> from itertools import product
>>> s = ' 11 2 333 4 '
>>> split(s, None)
['11', '2', '333', '4']
>>> n = 8
>>> for s in product('ab ', repeat=n):
... for maxsplit in range(-2, len(s)+2):
... s = ''.join(s)
... assert s.split(None, maxsplit) == split(s, None, maxsplit), namedtuple('Err', 'str maxsplit result target')(repr(s),
maxsplit, split(s,None,maxsplit), s.split(None, maxsplit))
"""
result = []
spmode = True
start = 0
if maxsplit != 0:
for i, c in enumerate(s):
if spmode:
if not c.isspace():
start = i
spmode = False
elif c.isspace():
result.append(s[start:i])
start = i
spmode = True
if len(result) == maxsplit:
break
rest = s[start:].lstrip()
return (result + [rest]) if rest else result
Once I have the cleanest possible, self-explantory code that passes tests, I'll improve the variable names and make a more sensible
docstring with readable examples. Surprisingly, it hasn't been a trivial exercise to come-up with an equivalent that corresponds
more closely to the way we think instead of corresponding the C code -- I want to show *what* is does more than *how* it does it.
Raymond
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documentation idea
On Oct 16, 2008, at 5:11 PM, Raymond Hettinger wrote: Raymond Hettinger wrote: * It will assist pypy style projects and other python implementations when they have to build equivalents to CPython. * Will eliminate confusion about what functions were exactly intended to do. * Will confer benefits similar to test driven development where the documentation and pure python version are developed first and doctests gotten to pass, then the C version is created to match. I haven't seen anyone comment about this assertion of "equivalence". Doesn't it strike you as difficult to maintain *two* versions of every function, and ensure they match *exactly*? Glad you brought this up. My idea is to present rough equivalence in unoptimized python that is simple and clear. The goal is to provide better documentation where code is more precise than English prose. That being said, some subset of the existing tests should be runnable against the rough equivalent and the python code should incorporate doctests. Running both sets of test should suffice to maintain the rough equivalence. This seems like a large undertaking. I'm sure you're not underestimating the effort, but I have the sense that you may be overestimating the usefulness of the results (or maybe I'm underestimating them through some lack of understanding). Would it be more optimal (in terms of both effort and results) to extend the existing documentation and/or docstrings with examples that use all of the functions so developers can see how to call them and what results to expect? Doug ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documentation idea
From: "Doug Hellmann" <[EMAIL PROTECTED] This seems like a large undertaking. Not necessarily. It can be done incrementally, starting with things like str.split() that almost no one understands completely. It should be put here and there where it adds some clarity. I'm sure you're not underestimating the effort, but I have the sense that you may be overestimating the usefulness of the results (or maybe I'm underestimating them through some lack of understanding). Would it be more optimal (in terms of both effort and results) to extend the existing documentation and/or docstrings with examples that use all of the functions so developers can see how to call them and what results to expect? The idea includes pure python code augmented by doctestable doctrings with enough examples. So, we're almost talking about the same thing. There is one difference; since the new attribute is guaranteed to be executable, it can be reliably run through doctest. The same is *not* true for arbitrary docstrings. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
Skip Montanaro wrote: > (Note: I'm backporting because the Python 2.6 version appears to be much more > robust than the 0.52 third-party release.) As Jesse points out, some of that robustness comes from long-standing bugs in the core getting fixed as a result of the addition of the multiprocessing unit tests to the standard library test suite. Not trying to discourage the project, just pointing out that it may not be as effective as hoped without patching the older versions of the interpreter. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Backporting multiprocessing?
Nick Coghlan wrote: As Jesse points out, some of that robustness comes from long-standing bugs in the core getting fixed as a result of the addition of the multiprocessing unit tests to the standard library test suite. Not trying to discourage the project, just pointing out that it may not be as effective as hoped without patching the older versions of the interpreter. Oh h... Are you able to recall a list of the most important bug fixes? Maybe we can get the bug fixes into 2.5.3 before it's too late. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
