Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Sun, Apr 17, 2011 at 4:19 AM, Raymond Hettinger wrote: > > On Apr 16, 2011, at 2:45 PM, Brett Cannon wrote: > > > On Sat, Apr 16, 2011 at 14:23, Stefan Krah wrote: >> >> Brett Cannon wrote: >> > In the grand python-dev tradition of "silence means acceptance", I >> > consider >> > this PEP finalized and implicitly accepted. > > I haven't seen any responses that said, yes this is a well thought-out > proposal that will actually benefit any of the various implementations. > Almost none of the concerns that have been raised has been addressed. Does > the PEP only apply to purely algorithmic modules such as heapq or does it > apply to anything written in C (like an xz compressor or for example)? My understanding is it does apply only to stuff that does not wrap an external library. > Does > testing every branch in a given implementation now guarantee every > implementation detail or do we only promise the published API (historically, > we've *always* done the latter)? Is there going to be any guidance on the > commonly encountered semantic differences between C modules and their Python > counterparts (thread-safety, argument handling, tracebacks, all possible > exceptions, monkey-patchable pure python classes versus hard-wired C types > etc)? > The PEP seems to be predicated on a notion that anything written in C is bad > and that all testing is good. Sounds about right > AFAICT, it doesn't provide any practical > advice to someone pursuing a non-trivial project (such as decimal or > threading). The PEP mostly seems to be about discouraging any further work > in C. If that's the case, it should just come out and say it rather than > tangentially introducing ambiguous testing requirements that don't make a > lot of sense. > The PEP also makes some unsupported claims about saving labor. My > understanding is the IronPython and Jython tend to re-implement modules > using native constructs. Even with PyPy, the usual pure python idioms > aren't necessarily what is best for PyPy, so I expect some rewriting there > also. We try very hard to optimize for usual python idioms. They're very often much better than specific cpython hacks. Unless you mean things like rebiding a global into default a "pythonic idiom". We had to rewrite places in standard library which are precisely not very pythonic. > It seems the lion's share of the work in making other implementations > has to do with interpreter details and whatnot -- I would be surprised if > the likes of bisect or heapq took even one-tenth of one percent of the total > development time for any of the other implementations. You're wrong. We didn't even write _heapq and _bisect. That's actually a lot of work and PyPy's team is quite small *and* it has to do all the other stuff as well. heapq and bisect were never a problem (except one case in twisted), but other stuff where C version diverged from Python version were a problem. Hell, we even wrote cPickle which wraps pickle and provides correct interface! This is kind of things we would rather not spend time on (and yes, it is time consuming). > >> >> I did not really see an answer to these concerns: >> >> http://mail.python.org/pipermail/python-dev/2011-April/110672.html > > Antoine does seem sold on the 100% branch coverage requirement and views it > as pointless. I disagree. =) > > As for the exception Stefan is saying may be granted, that is not in the PEP > so I consider it unimportant. If we really feel the desire to grant an > exception we can (since we can break any of our own rules that we > collectively choose to), but I'm assuming we won't. > >> >> http://mail.python.org/pipermail/python-dev/2011-April/110675.html > > Raymond thinks that have a testing requirement conflates having > implementations match vs. APIs. > > That is not an accurate restatement of my post. > > Well, as we all know, the stdlib ends up having its implementation details > relied upon constantly by people whether they mean to or not, so making > sure that this is properly tested helps deal with this known reality. > > If you're saying that all implementation details (including internal > branching logic) are now guaranteed behaviors, then I think this PEP has > completely lost its way. I don't know of any implementors asking for this. > > This is a damned-if-you-do-damned-if-you-don't situation. The first draft of > this PEP said to be "semantically equivalent w/ divergence where technically > required", but I got pushback from being too wishy-washy w/ lack of concrete > details. So I introduce a concrete metric that some are accusing of being > inaccurate for the goals of the PEP. I'm screwed or I'm screwed. =) So I am > choosing to go with the one that has a side benefit of also increasing test > coverage. > > Maybe that is just an indication that the proposal isn't mature yet. To > me, it doesn't seem well thought out and isn't realistic. > > Now if people would actually support simply not accepting any more C m
Re: [Python-Dev] Status of json (simplejson) in cpython
Nick Coghlan gmail.com> writes: > Consider trying: > > import sys > sys.modules["_json"] = 0 # Block the C extension > import json > > in a fresh interpreter. > Thanks for the tip. The revised script at https://gist.github.com/924626 shows more believable numbers vis-à-vis the no-speedups case. Interestingly this morning, stdlib json wins in both cases, though undoubtedly YMMV. --- (jst3)vinay@eta-natty:~/projects/scratch$ python time_json.py --no-speedups Python version: 3.2 (r32:88445, Mar 25 2011, 19:28:28) [GCC 4.5.2] 11.21484375 KiB read Timing simplejson (without speedups): 4.585145950317383 Timing stdlib json (without speedups): 3.9949100017547607 (jst3)vinay@eta-natty:~/projects/scratch$ python time_json.py Python version: 3.2 (r32:88445, Mar 25 2011, 19:28:28) [GCC 4.5.2] 11.21484375 KiB read Timing simplejson (with speedups): 0.3202629089355469 Timing stdlib json (with speedups): 0.3200039863586426 --- Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On 18 April 2011 08:05, Maciej Fijalkowski wrote: > On Sun, Apr 17, 2011 at 4:19 AM, Raymond Hettinger > wrote: >> Almost none of the concerns that have been raised has been addressed. Does >> the PEP only apply to purely algorithmic modules such as heapq or does it >> apply to anything written in C (like an xz compressor or for example)? > > My understanding is it does apply only to stuff that does not wrap an > external library. My understanding is that this is most people's understanding, so it should be explicitly documented in the PEP. It would also be worth asking: are there any other reasons for using C code beyond wrapping external libraries and accelerating code that could equally be written in Python? I can't think of any, myself, but OTOH I wonder if the *degree* of acceleration is also relevant - some things (compression algorithms, for example) just can't realistically be coded in Python (CPython, at least). >> The PEP seems to be predicated on a notion that anything written in C is bad >> and that all testing is good. > > Sounds about right I disagree. To me, a Python without libraries such as os, zlib, zipfile, threading, etc wouldn't be much use (except in specialised circumstances). OK, that means that alternative implementations need to do extra work to implement equivalents in their own low-level language, but so be it (sorry!) This PEP has a flavour to me of the old "100% pure Java" ideal, where Java coders expected everything to be reimplemented in Java, avoiding any native code. I didn't like the idea then, and I don't have much more love for it now in Python. (OK, I know this is an exaggeration of the position the PEP is taking, but without more clarity in the PEP's language, I honestly don't know how much of an exaggeration). Maybe the PEP could go through the various C libraries in the stdlib at the moment, and discuss how the PEP would address them? It would be useful to see how much of an impact the PEP would have had if it had been Python policy from the start... Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Mon, 18 Apr 2011 09:36:20 +0100 Paul Moore wrote: > On 18 April 2011 08:05, Maciej Fijalkowski wrote: > > On Sun, Apr 17, 2011 at 4:19 AM, Raymond Hettinger > > wrote: > > >> Almost none of the concerns that have been raised has been addressed. Does > >> the PEP only apply to purely algorithmic modules such as heapq or does it > >> apply to anything written in C (like an xz compressor or for example)? > > > > My understanding is it does apply only to stuff that does not wrap an > > external library. > > My understanding is that this is most people's understanding, so it > should be explicitly documented in the PEP. > > It would also be worth asking: are there any other reasons for using C > code beyond wrapping external libraries and accelerating code that > could equally be written in Python? faulthandler is an example. Very low-level tinkering with threads, signal handlers and possibly corrupt memory simply can't be done in Python. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Mon, 18 Apr 2011 09:36:20 +0100, Paul Moore wrote: > On 18 April 2011 08:05, Maciej Fijalkowski wrote: > > On Sun, Apr 17, 2011 at 4:19 AM, Raymond Hettinger > > wrote: > > >> The PEP seems to be predicated on a notion that anything written in C is > >> bad and that all testing is good. > > > > Sounds about right > > I disagree. To me, a Python without libraries such as os, zlib, > zipfile, threading, etc wouldn't be much use (except in specialised > circumstances). OK, that means that alternative implementations need > to do extra work to implement equivalents in their own low-level > language, but so be it (sorry!) I think Maciej left out an "only" in that sentence. If you say "only C", then the sentence makes sense, even when applied to modules that *can* only be written in C (for CPython). That is, not having a Python version is bad. Necessary in many cases (or not worth the cost, for external library wrappers), but wouldn't it be nicer if it wasn't necessary? > This PEP has a flavour to me of the old "100% pure Java" ideal, where > Java coders expected everything to be reimplemented in Java, avoiding > any native code. I didn't like the idea then, and I don't have much > more love for it now in Python. (OK, I know this is an exaggeration of > the position the PEP is taking, but without more clarity in the PEP's > language, I honestly don't know how much of an exaggeration). The Pythonic ideal contains quite a bit of pragmatism, so yes, that is an exaggeration of the goals of the PEP, certainly. (Although pypy may do it anyway, for pragmatic reasons :) > Maybe the PEP could go through the various C libraries in the stdlib > at the moment, and discuss how the PEP would address them? It would be > useful to see how much of an impact the PEP would have had if it had > been Python policy from the start... That might indeed be a useful exercise, especially since other implementations (or even perhaps CPython developers) may want to contribute Python-only versions and/or tests for things that would have been affected by the PEP. I don't have time to do it right now, but if I can pry any time loose I'll have it near the top of my list. -- R. David Murray http://www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 2.6.7
With Martin getting ready to release 2.5.6, I think it's time to prepare a 2.6.7 source-only security release. I'll work my way through the NEWS file and recent commits, but if there is anything that you know is missing from the 2.6 branch, please let me know. It would be especially helpful if there were bugs for any such issues. Cheers, -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Forward of moderated message
--- Begin Message --- Hi Martin, On Sun, 17 Apr 2011 23:57:05 +0200 "Martin v. Löwis" wrote: > On behalf of the Python development team and the Python community, I'm > happy to announce the release candidate 1 of Python 2.5.6. > > This is a source-only release that only includes security fixes. The > last full bug-fix release of Python 2.5 was Python 2.5.4. Users are > encouraged to upgrade to the latest release of Python 2.7 (which is > 2.7.1 at this point). Maybe in future the last sentence could be changed to something like: Users are encouraged to upgrade to the latest release of Python 2.7 (which is 2.7.1 at this point), or better still, to migrate to Python 3.2. (Naturally changing the versions as appropriate.) -- Mark Summerfield, Qtrac Ltd, www.qtrac.eu C++, Python, Qt, PyQt - training and consultancy "Advanced Qt Programming" - ISBN 0321635906 http://www.qtrac.eu/aqpbook.html --- End Message --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Post from Mark Summerfield
Mark Summerfield responded to Martin's python-announce post. Rather than approving it I rejected it and forwarded it here. (I suppose I could have forwarded it directly to Martin, but that would have required that I recall or look up his email address...) Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
Hi, We try very hard to optimize for usual python idioms. They're very often much better than specific cpython hacks. Unless you mean things like rebiding a global into default a "pythonic idiom". We had to rewrite places in standard library which are precisely not very pythonic. If I understand correctly, you’ve made internal changes preserving the official API of the modules. Have you reported those cases to bugs.python.org? I’m sure we’d be glad to incorporate those changes into the stdlib, possibly even in the stable branches if their rationale is strong enough. Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
Hi, Perhaps we need a @python_implementation_detail skip decorator? That’s called test.support.cpython_only (see also test.support.check_impl_detail). You’re welcome. Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Mon, Apr 18, 2011 at 6:32 PM, Éric Araujo wrote: > Hi, > >> We try very hard to optimize for usual python idioms. They're very >> often much better than specific cpython hacks. Unless you mean things >> like rebiding a global into default a "pythonic idiom". We had to >> rewrite places in standard library which are precisely not very >> pythonic. > > If I understand correctly, you’ve made internal changes preserving the > official API of the modules. Have you reported those cases to > bugs.python.org? I’m sure we’d be glad to incorporate those changes > into the stdlib, possibly even in the stable branches if their rationale > is strong enough. I think what's relevant was merged by benjamin. Usually: * we do revert things that were specifically made to make cpython faster, like def f(_getattr=getattr): ... * we usually target CPython version that's already frozen, which is pretty inconvinient to post this changes back. Example would be a socket module where it has changed enough in 3.x that 2.7 changes make no sense. > > Regards > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Apr 18, 2011, at 10:11 AM, Maciej Fijalkowski wrote: > > * we usually target CPython version that's already frozen, which is > pretty inconvinient to post this changes back. Example would be a > socket module where it has changed enough in 3.x that 2.7 changes make > no sense. Do you have any thoughts on the problem with the concrete C API not working well with subclasses of builtin types? I'm thinking that the PEP should specifically ban the practice of using the concrete api unless it is known for sure that an object is an exact type match. It is okay to write PyList_New() followed by PyList_SetItem() but not okay to use PyList_SetItem() on a user supplied argument that is known to be a subclass of list. A fast path can be provided for an exact path, but there would also need to a be a slower path that either converts the object to an exact list or that uses PyObject_SetItem(). In the discussions about this topic, there doesn't seem to be any technical solutions; instead, it will take a social solution such as a PEP and clear warnings in the docs. Raymond___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
On Mon, 18 Apr 2011 18:34:06 +0200, =?UTF-8?Q?=C3=89ric_Araujo?= wrote: > > Perhaps we need a @python_implementation_detail skip decorator? > Thatâs called test.support.cpython_only (see also > test.support.check_impl_detail). Youâre welcome. Nope. That's not what I was talking about. I was talking about marking a test as something that we require only the *python* implementation of a module to pass (presumably because it tests an internal implementation detail). Thus a C accelerator would not be expected to pass that test, nor would a C# accelerator, but pypy or any platform without an accelerator (that is, anything *using* the python code) would be expected to pass it. I would hope that such tests would be vanishingly rare (that is, that all needed tests can be expressed as black box tests). -- R. David Murray http://www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: #11731: simplify/enhance parser/generator API by introducing policy objects.
On 18.04.2011 20:00, r.david.murray wrote:
> diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst
> --- a/Doc/library/email.parser.rst
> +++ b/Doc/library/email.parser.rst
> @@ -112,8 +118,13 @@
> :class:`~email.message.Message` (see :mod:`email.message`). The factory
> will
> be called without arguments.
>
> - .. versionchanged:: 3.2
> - Removed the *strict* argument that was deprecated in 2.4.
> + The *policy* keyword specifies a :mod:`~email.policy` object that
> controls a
> + number of aspects of the parser's operation. The default policy maintains
> + backward compatibility.
> +
> + .. versionchanged:: 3.3
> + Removed the *strict* argument that was deprecated in 2.4. Added the
> + *policy* keyword.
Hmm, so *strict* wasn't actually removed in 3.2?
> @@ -187,12 +204,15 @@
>
> .. currentmodule:: email
>
> -.. function:: message_from_string(s, _class=email.message.Message,
> strict=None)
> +.. function:: message_from_string(s, _class=email.message.Message, *, \
> + policy=policy.default)
>
> Return a message object structure from a string. This is exactly
> equivalent to
> - ``Parser().parsestr(s)``. Optional *_class* and *strict* are interpreted
> as
> + ``Parser().parsestr(s)``. *_class* and *policy* are interpreted as
> with the :class:`Parser` class constructor.
>
> + .. versionchanged:: removed *strict*, added *policy*
> +
The 3.3 version is missing here. Also, please always end version directive text
with a period.
> .. function:: message_from_bytes(s, _class=email.message.Message,
> strict=None)
>
> Return a message object structure from a byte string. This is exactly
> @@ -200,21 +220,27 @@
> *strict* are interpreted as with the :class:`Parser` class constructor.
>
> .. versionadded:: 3.2
> + .. versionchanged:: 3.3 removed *strict*, added *policy*
See above.
> -.. function:: message_from_file(fp, _class=email.message.Message,
> strict=None)
> +.. function:: message_from_file(fp, _class=email.message.Message, *, \
> +policy=policy.default)
>
> Return a message object structure tree from an open :term:`file object`.
> - This is exactly equivalent to ``Parser().parse(fp)``. Optional *_class*
> - and *strict* are interpreted as with the :class:`Parser` class
> constructor.
> + This is exactly equivalent to ``Parser().parse(fp)``. *_class*
> + and *policy* are interpreted as with the :class:`Parser` class
> constructor.
>
> -.. function:: message_from_binary_file(fp, _class=email.message.Message,
> strict=None)
> + .. versionchanged:: 3.3 removed *strict*, added *policy*
See above.
> +.. function:: message_from_binary_file(fp, _class=email.message.Message, *, \
> + policy=policy.default)
>
> Return a message object structure tree from an open binary :term:`file
> object`. This is exactly equivalent to ``BytesParser().parse(fp)``.
> - Optional *_class* and *strict* are interpreted as with the :class:`Parser`
> + *_class* and *policy* are interpreted as with the :class:`Parser`
> class constructor.
>
> .. versionadded:: 3.2
> + .. versionchanged:: 3.3 removed *strict*, added *policy*
See above.
> --- /dev/null
> +++ b/Doc/library/email.policy.rst
> @@ -0,0 +1,179 @@
> +:mod:`email`: Policy Objects
> +
> +
> +.. module:: email.policy
> + :synopsis: Controlling the parsing and generating of messages
This file should have a ".. versionadded:: 3.3" (without further content) here.
> +The :mod:`email` package's prime focus is the handling of email messages as
> +described by the various email and MIME RFCs. However, the general format of
> +email messages (a block of header fields each consisting of a name followed
> by
> +a colon followed by a value, the whole block followed by a blank line and an
> +arbitrary 'body'), is a format that has found utility outside of the realm of
> +email. Some of these uses conform fairly closely to the main RFCs, some do
> +not. And even when working with email, there are times when it is desirable
> to
> +break strict compliance with the RFCs.
> +
> +Policy objects are the mechanism used to provide the email package with the
> +flexibility to handle all these disparate use cases,
Looks like something is missing from this sentence :)
[...]
> +As an example, the following code could be used to read an email message
> from a
> +file on disk and pass it to the system ``sendmail`` program on a ``unix``
> +system::
Should be Unix, not ``unix``.
> + >>> from email import msg_from_binary_file
> + >>> from email.generator import BytesGenerator
> + >>> import email.policy
> + >>> from subprocess import Popen, PIPE
> + >>> with open('mymsg.txt', 'b') as f:
> + >>> msg = msg_from_binary_file(f, policy=email.policy.mbox)
> + >>> p = Popen(['sendmail', msg[
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
Maciej Fijalkowski, 18.04.2011 19:11: On Mon, Apr 18, 2011 at 6:32 PM, Éric Araujo wrote: We try very hard to optimize for usual python idioms. They're very often much better than specific cpython hacks. Unless you mean things like rebiding a global into default a "pythonic idiom". We had to rewrite places in standard library which are precisely not very pythonic. If I understand correctly, you’ve made internal changes preserving the official API of the modules. Have you reported those cases to bugs.python.org? I’m sure we’d be glad to incorporate those changes into the stdlib, possibly even in the stable branches if their rationale is strong enough. I think what's relevant was merged by benjamin. Usually: * we do revert things that were specifically made to make cpython faster, like def f(_getattr=getattr): ... Thanks. Speaking for the Cython project, we are certainly happy to see these micro optimisations reverted. Makes our life easier and the generated code faster. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: #11731: simplify/enhance parser/generator API by introducing policy objects.
On Mon, 18 Apr 2011 20:26:36 +0200, Georg Brandl wrote:
> On 18.04.2011 20:00, r.david.murray wrote:
>
> > diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst
> > --- a/Doc/library/email.parser.rst
> > +++ b/Doc/library/email.parser.rst
> > @@ -112,8 +118,13 @@
> > :class:`~email.message.Message` (see :mod:`email.message`). The
> > factory will
> > be called without arguments.
> >
> > - .. versionchanged:: 3.2
> > - Removed the *strict* argument that was deprecated in 2.4.
> > + The *policy* keyword specifies a :mod:`~email.policy` object that
> > controls a
> > + number of aspects of the parser's operation. The default policy
> > maintains
> > + backward compatibility.
> > +
> > + .. versionchanged:: 3.3
> > + Removed the *strict* argument that was deprecated in 2.4. Added the
> > + *policy* keyword.
>
> Hmm, so *strict* wasn't actually removed in 3.2?
Correct. I had previously checked in a versionchanged with the wrong
release number. Should have corrected that separately.
[...]
> > +Policy objects are the mechanism used to provide the email package with the
> > +flexibility to handle all these disparate use cases,
>
> Looks like something is missing from this sentence :)
Ãric thought so too, but it reads fine to me. Maybe it is colloquial
grammar and I'm just blind to it. I can't now remember what his suggested
modification was, either. I've rewritten it as:
Policy objects give the email package the flexibility to handle all
these disparate use cases.
[...]
> > + >>> from email import msg_from_binary_file
> > + >>> from email.generator import BytesGenerator
> > + >>> import email.policy
> > + >>> from subprocess import Popen, PIPE
> > + >>> with open('mymsg.txt', 'b') as f:
> > + >>> msg = msg_from_binary_file(f, policy=email.policy.mbox)
> > + >>> p = Popen(['sendmail', msg['To'][0].address], stdin=PIPE)
> > + >>> g = BytesGenerator(p.stdin, email.policy.policy=SMTP)
>
> That keyword arg doesn't look right.
Yep, I got that backward when I edited it.
> > + >>> g.flatten(msg)
> > + >>> p.stdin.close()
> > + >>> rc = p.wait()
>
> Also, if you put interactive prompts, please use them correctly ("..." prompt
> and one blank line for the with block).
Ãric had me fix one example of that already. What do you mean by "one
blank line" though? Doctest doesn't require blank lines after the ...
lines, does it?
[...]
> > + .. method:: handle_defect(obj, defect)
> > +
> > + *obj* is the object on which to register the defect.
>
> What kind of object is *obj*?
Whatever object is being used to represent the data being parsed when
the defect is found. Right now that's always a Message, but that won't
continue to be true. The rest of the documentation mentions or will
mention which objects have defect lists, and it felt like duplicating
that information here was a bad case of repetition (ie: one that doesn't
add much value compared to the danger of it getting out of date.)
Except for this last item, I've fixed everything and will shortly
check it in.
> Sorry for the long review.
It was a long patch. No need to apologize.
--
R. David Murray http://www.bitdance.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
Raymond Hettinger, 18.04.2011 19:26: On Apr 18, 2011, at 10:11 AM, Maciej Fijalkowski wrote: * we usually target CPython version that's already frozen, which is pretty inconvinient to post this changes back. Example would be a socket module where it has changed enough in 3.x that 2.7 changes make no sense. Do you have any thoughts on the problem with the concrete C API not working well with subclasses of builtin types? I'm thinking that the PEP should specifically ban the practice of using the concrete api unless it is known for sure that an object is an exact type match. Absolutely. It is okay to write PyList_New() followed by PyList_SetItem() but not okay to use PyList_SetItem() on a user supplied argument that is known to be a subclass of list. A fast path can be provided for an exact path, but there would also need to a be a slower path that either converts the object to an exact list or that uses PyObject_SetItem(). For what it's worth, Cython generates code that contains optimistic optimisations for common cases, such as iteration, x.append() calls, etc. When it finds such a pattern, it generates separate code paths for the most likely (builtin type) case and a slower fallback for the more unlikely case of a user provided type. So you get both speed and compatibility for free, just by writing idiomatic code like "for item in some_iterable". Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
R. David Murray, 18.04.2011 14:30: On Mon, 18 Apr 2011 09:36:20 +0100, Paul Moore wrote: On 18 April 2011 08:05, Maciej Fijalkowski wrote: On Sun, Apr 17, 2011 at 4:19 AM, Raymond Hettinger wrote: The PEP seems to be predicated on a notion that anything written in C is bad and that all testing is good. Sounds about right I disagree. To me, a Python without libraries such as os, zlib, zipfile, threading, etc wouldn't be much use (except in specialised circumstances). OK, that means that alternative implementations need to do extra work to implement equivalents in their own low-level language, but so be it (sorry!) I think Maciej left out an "only" in that sentence. If you say "only C", then the sentence makes sense, even when applied to modules that *can* only be written in C (for CPython). That is, not having a Python version is bad. Necessary in many cases (or not worth the cost, for external library wrappers), but wouldn't it be nicer if it wasn't necessary? FWIW, there is a proposed GSoC project that aims to implement a Cython backend for PyPy, either using ctypes or PyPy's own FFI. That would basically remove the need to write library wrappers in C for both CPython and PyPy, and eventually for IronPython, which also has a Cython port in the making. Not sure how Jython fits into this, but I wouldn't object to someone writing a JNI backend either. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
I just want to say upfront that my personal life has just gotten very hectic as of late (green card stuff for my wife who is Canadian) and probably will not let up until June. So if I go a while without replying to points being made for quite a while, I apologize. Luckily there seem to be others here who understand the direction I am coming from so there is no need to stop talking while I am pre-occupied with the real world. On Sun, Apr 17, 2011 at 00:30, Raymond Hettinger < [email protected]> wrote: > > In the grand python-dev tradition of "silence means acceptance", I > consider > this PEP finalized and implicitly accepted. > >> > >> I haven't seen any responses that said, yes this is a well thought-out > proposal > >> that will actually benefit any of the various implementations. > > > > In that case it may well be that the silence is because the other > > implementations think the PEP is OK. They certainly voted in favor of > > the broad outline of it at the language summit. > > Sounds like it was implicitly accepted even before it was written or any of > the details were discussed. > Actually I directly emailed the relevant people from the other VMs to make sure they were happy with what I was aiming for before I approached python-dev with the PEP. So IronPython, Jython, and PyPy lead developers have all told me that they want something along the lines of this PEP to happen. > > The big picture of "let's do something to make life easier for other > implementations" is a worthy goal. What that something should be is still a > bit ambiguous. > > > >> every branch in a given implementation now guarantee every > implementation detail > >> or do we only promise the published API (historically, we've *always* > done the > >> latter)? > > > > As Brett said, people do come to depend on the details of the > > implementation. But IMO the PEP should be clarified to say that the > > tests we are talking about should be tests *of the published API*. > > That is, blackbox tests, not whitebox tests. > > +1 That's an excellent suggestion. Without that change, it seems like the > PEP is overreaching. > I'm okay with going with this line of through, including R. David's "100% branch coverage is but one way to achieve extensive testing of the published API". > > > >> Is there going to be any guidance on the commonly encountered semantic > >> differences between C modules and their Python counterparts > (thread-safety, > >> argument handling, tracebacks, all possible exceptions, monkey-patchable > pure > >> python classes versus hard-wired C types etc)? > > > > Presumably we will need to develop such guidance. > > +1 That would be very helpful. Right now, the PEP doesn't address any of > the commonly encountered differences. > If people are willing to help me (i.e., go ahead and edit the PEP) with this then I am okay with adding some common issues (but I don't expect it to be exhaustive). > > > > I personally have no problem with the 100% coverage being made a > > recommendation in the PEP rather than a requirement. It sounds like > > that might be acceptable to Antoine. Actually, I would also be fine with > > saying "comprehensive" instead, with a note that 100% branch coverage is > > a good way to head toward that goal, since a comprehensive test suite > > should contain more tests than the minimum set needed to get to 100% > > branch coverage. > > +1 better test coverage is always a good thing (IMO). > > > Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
Brett Cannon, 05.04.2011 01:46: At both the VM and language summits at PyCon this year, the issue of compatibility of the stdlib amongst the various VMs came up. Two issues came about in regards to modules that use C code. One is that code that comes in only as C code sucks for all other VMs that are not CPython since they all end up having to re-implement that module themselves. Two is that modules that have an accelerator module (e.g., heapq, warnings, etc.) can end up with compatibility options (sorry, Raymond, for picking on heapq, but is was what bit the PyPy people most recently =). In lieu of all of this, here is a draft PEP to more clearly state the policy for the stdlib when it comes to C code. Since this has come up before and this was discussed so much at the summits I have gone ahead and checked this in so that even if this PEP gets rejected there can be a written record as to why. And before anyone asks, I have already run this past the lead devs of PyPy, Jython, and IronPython and they all support what this PEP proposes. And with the devs of the other VMs gaining push privileges there shouldn't be an added developer burden on everyone to make this PEP happen. This PEP has received a lengthy discussion by now, so here's why I think it's being fought so heavily by several CPython core developers, specifically those who have traditionally carried a large part of the optimisation load in the project. I think the whole point of this PEP is that, having agreed that a shared standard library for all Python implementations is a good thing, the amount of shareable code should be maximised. I doubt that anyone will argue against this goal. But that obviously includes all sides. If other implementations are free to cherry pick the targets of their own effort geared by the optimisation of their own implementation, and leave the whole burden of compatibility and code reusability on CPython, in addition to the CPython efforts of improving and optimising its own core code base and its own stdlib version, it's not an equal matter. That's what makes the PEP feel so unfair to CPython developers, because they are the ones who carry most of the burden of maintaining the stdlib in the first place, and who will most likely continue to carry it, because other implementations will continue to be occupied with their own core development for another while or two. It is nice to read that other implementations are contributing back patches that simplify their own reuse of the stdlib code. However, that does not yet make them equal contributors to the development and the maintenance of the stdlib, and is of very little worth to the CPython project. It often even runs counter to the interest of CPython itself. I think this social problem of the PEP can only be solved if the CPython project stops doing the major share of the stdlib maintenance, thus freeing its own developer capacities to focus on CPython related improvements and optimisations, just like the other implementations currently do. I'm not sure we want that at this point. Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: #11731: simplify/enhance parser/generator API by introducing policy objects.
On 18.04.2011 21:39, R. David Murray wrote:
>> > +Policy objects are the mechanism used to provide the email package with
>> > the
>> > +flexibility to handle all these disparate use cases,
>>
>> Looks like something is missing from this sentence :)
>
> Éric thought so too, but it reads fine to me. Maybe it is colloquial
> grammar and I'm just blind to it. I can't now remember what his suggested
> modification was, either. I've rewritten it as:
>
> Policy objects give the email package the flexibility to handle all
> these disparate use cases.
Sure, I was only asking because the original ended in a trailing comma.
>> > + >>> from email import msg_from_binary_file
>> > + >>> from email.generator import BytesGenerator
>> > + >>> import email.policy
>> > + >>> from subprocess import Popen, PIPE
>> > + >>> with open('mymsg.txt', 'b') as f:
>> > + >>> msg = msg_from_binary_file(f, policy=email.policy.mbox)
>> > + >>> p = Popen(['sendmail', msg['To'][0].address], stdin=PIPE)
>> > + >>> g = BytesGenerator(p.stdin, email.policy.policy=SMTP)
>>
>> That keyword arg doesn't look right.
>
> Yep, I got that backward when I edited it.
>
>> > + >>> g.flatten(msg)
>> > + >>> p.stdin.close()
>> > + >>> rc = p.wait()
>>
>> Also, if you put interactive prompts, please use them correctly ("..." prompt
>> and one blank line for the with block).
>
> Éric had me fix one example of that already. What do you mean by "one
> blank line" though? Doctest doesn't require blank lines after the ...
> lines, does it?
Not sure what doctest requires, but in the actual interactive shell you'd have
>>> with ...:
... do something
...
>>> next statement
It's not really important though.
Georg
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
