Re: [Python-Dev] Language Summit Follow-Up

[Nick Coghlan]

On 29 May 2014 08:26, Glyph Lefkowitz  wrote:

Thanks for the write-up!

> Here are some ideas for Python 2.7+.
>
> Add ensurepip to the installers.  Having pip reliably available increases
> the availability of libraries that help with porting, and will generally
> strengthen the broader ecosystem in the (increasingly long) transition
> period.

Agreed on this one - I only dropped it from 453 because I didn't think
I could continue to push for it and still make the beta deadline for
3.4, not because I changed my mind. This mainly needs a champion to
write the PEP to describe the exact details of what would be
backported (the tricky parts of PEP 453 relating to pyvenv don't
apply) and to actually do the work. Bundling the Python Launcher with
the Windows installer would also be desirable.

> Add some warnings about python 3 compatibility.
>
> It should at least be possible to get a warning for every single implicit
> string coercion.
> Old-style classes.
> Old-style division.
> Print statements.
> Old-style exception syntax.
> buffer().
> bytes(memoryview(b'abc'))
> Importing old locations from the stdlib (see point 4.)
> Long integer syntax.
> Use of variables beyond the lifetime of an 'except Exception as e' block or
> a list comprehension.

A few more for the "missing -3 warning" list:

- calling str.encode
- calling unicode.decode
- returning str from str.decode
- returning unicode from unicode.encode

The relevant TypeErrors in 3.x show where the latter two warnings need
to go in 2.7.

More controversially: warn for any calls to Py2 str methods that
aren't present on Py3 bytes.

> Backport 'yield from' to allow people to use Tulip and Tulip-compatible
> code, and to facilitate the development of Tulip-friendly libraries and a
> Tulip ecosystem.  A robust Tulip ecosystem requires the participation of
> people who are not yet using Python 3.

Given that the target audience here is our more conservative user
base, Tulip/Trollius backends for Twisted and gevent seem more
conducive to that than supporting Tulip's native coroutine syntax.
That said, it's a pure additive change without any significant
backwards compatibility implications, so I personally wouldn't object
if Guido said yes.

> Add aliases for the renamed modules in the stdlib.  This will allow people
> to "just write python 3" in a lot more circumstances.

This is what the module aliasing options provided by python-future are
designed to do - it's much closer to "just write Python 3" than six is
(although that comes at the price of being a bit more magical).

The python-future experience also shows there's a potential backwards
compatibility issue here - depending on exactly how you do it, you can
end up conflicting with other libraries trying to do the same thing.

> (re-)Enable warnings by default, including enabling -3 warnings.  Right now
> all warnings are silent by default, which greatly reduces discoverability of
> future compatibility issues.  I hope it's not controversial to say that most
> new Python code is still being written against Python 2.7 today; if people
> are writing that code in such a way that it's not 3-friendly, it should be a
> more immediately noticeable issue.

The reasons these were turned off haven't changed, and passing -3
already enables DeprecationWarnings generally (along with -Qwarn).
However, several of the warnings mentioned in the list above are
currently missing entirely, so they won't show up regardless of the
warning state.

> Get rid of 2to3. Particularly, of any discussion of using 2to3 in the
> documentation.  More than one very experienced, well-known Python developer
> in this discussion has told me that they thought 2to3 was the blessed way to
> port their code, and it's no surprise that they think so, given that the
> first technique  mentions is
> still 2to3.  We should replace 2to3 with something like
> . 2to3 breaks your code on
> python 2, and doesn't necessarily get it running on python 3.  A more
> conservative approach that reduced the amount of work to get your code 2/3
> compatible but was careful to leave everything working would be a lot more
> effective.

+1, although I think python-future provides a better "subset language"
experience than modernize+six does, and provides a 3->common subset
converter in addition to a 2->common subset (see
http://python-future.org/faq.html#relationship-between-python-future-and-other-compatibility-tools
for more on the difference between the two approaches).

The futurize script also has the nice property of supporting two
different stages, with "--stage1" being a pure syntactic update - no
new runtime dependencies needed.

"Big bang" migrations are highly unlikely to be the right choice for
anyone, and while the code produced by python-future will still have
some quirks (e.g. avoiding direct use of any mapping methods as PEP
469 ended up reco

[Python-Dev] [Python-checkins] Daily reference leaks (8391f99208f6): sum=181

[Brett Cannon]

I think the memory leak was caused by
http://hg.python.org/cpython/rev/7d20e30bd540 because
http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets
the 'res' variable and then overwrites it unconditionally w/o PY_DECREF
beforehand.

On Thu May 29 2014 at 4:02:17 AM,  wrote:

> results for 8391f99208f6 on branch "default"
> 
>
> test_collections leaked [0, 2, 4] references, sum=6
> test_collections leaked [0, 1, 2] memory blocks, sum=3
> test_functools leaked [0, 0, 3] memory blocks, sum=3
> test_importlib leaked [5, 5, 5] references, sum=15
> test_pkgutil leaked [1, 1, 1] references, sum=3
> test_site leaked [0, 0, 2] references, sum=2
> test_site leaked [0, 0, 2] memory blocks, sum=2
> test_zipimport leaked [47, 47, 47] references, sum=141
> test_zipimport_support leaked [2, 2, 2] references, sum=6
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x']
> ___
> Python-checkins mailing list
> python-check...@python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Language Summit Follow-Up

[Brett Cannon]

On Wed May 28 2014 at 10:14:39 PM, Glyph Lefkowitz 
wrote:

> At the language summit, Alex and I volunteered to put together some
> recommendations on what changes could be made to Python (the language) in
> order to facilitate a smoother transition from Python 2 to Python 3.  One
> of the things that motivated this was the (surprising, to us) consideration
> that features like ensurepip might be added to the future versions of the
> 2.7 installers from python.org.
>
> The specific motivations for writing this are:
>
>
>1. Library maintainers have a rapidly expanding matrix that requires
>an increasing number of branches to satisfy.
>2. People with large corporate codebases absolutely cannot port all at
>once.
>
>
> If you don't have perfect test coverage then you can't make any progress.
>  So these changes are intended to make porting from python 2 to python 3
> more guided and incremental.  We believe that these attributes are
> necessary.
>
> We would like to stress that we don't believe anything on this list is as
> important as the continuing efforts that everyone in the broader ecosystem
> is making.  If you just want to ease the transition by working on anything
> at all, the best use of your time right now is porting
> https://warehouse.python.org/project/MySQL-python/ to Python 3. :)
>
> Nevertheless there are some things that the language and CPython could do.
>
> Unfortunately we had to reject any proposal that involved new __future__
> imports, since unknown __future__ imports are un-catchable SyntaxErrors.
>
> Here are some ideas for Python 2.7+.
>
[SNIP]

>
>1. Get rid of 2to3. Particularly, of any discussion of using 2to3 in
>the documentation.  More than one very experienced, well-known Python
>developer in this discussion has told me that they thought 2to3 was the
>blessed way to port their code, and it's no surprise that they think so,
>given that the first technique <
>https://docs.python.org/3/howto/pyporting.html> mentions is still
>2to3.  We should replace 2to3 with something like <
>https://github.com/mitsuhiko/python-modernize>. 2to3 breaks your code
>on python 2, and doesn't necessarily get it running on python 3.  A more
>conservative approach that *reduced* the amount of work to get your
>code 2/3 compatible but was careful to leave everything working would be a
>lot more effective.
>
>
Two things. One is the HOWTO mentions 2to3 *only* when you are dropping
Python 3 support and since that makes your life the easiest it's listed as
the first option. You will notice point 2 in the TL;DR is be source
compatible if keeping Python 2 compatibility. The main part of the doc is
also all about source compatibility and using 2to3 while maintaining
compatibility is relegated to the Alternatives section at the end (heck,
2to3 is mentioned only 3 times in that entire document, one of which is in
a code comment).

Two, is how do you see python-future fit into this (I know others have
brought it up but we have two different approaches now to keeping source
compatibility that vary in magic and thus smoothing out edges)?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Language Summit Follow-Up

[Terry Reedy]

On 5/28/2014 6:26 PM, Glyph Lefkowitz wrote:


I hope it's
not controversial to say that most new Python code is still being
written against Python 2.7 today;


Given that Python 3 downloads now outnumber Python 2 downloads, I think 
'most' might be an overstatement. But I think it a moot point.


> if people are writing that code in such a way that it's not
> 3-friendly, it should be a more immediately noticeable issue.

If the truth were, conservatively, 1/4 of new Python code in 2.7, or 
even less, I would still be in favor of making 3-friendly 2.7 code easier.


This is also important for the separate codebase approach, as in the 
stdlib. Just last week, I got a rejected chunk when backporting because 
a 2.7 idlelib module uses 'file(...' instead of 'open(...'.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (8391f99208f6): sum=181

[Eric Snow]

Good catch.  I'll look into it.

-eric

On Thu, May 29, 2014 at 8:49 AM, Brett Cannon  wrote:
> I think the memory leak was caused by
> http://hg.python.org/cpython/rev/7d20e30bd540 because
> http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets the
> 'res' variable and then overwrites it unconditionally w/o PY_DECREF
> beforehand.
>
>
> On Thu May 29 2014 at 4:02:17 AM,  wrote:
>>
>> results for 8391f99208f6 on branch "default"
>> 
>>
>> test_collections leaked [0, 2, 4] references, sum=6
>> test_collections leaked [0, 1, 2] memory blocks, sum=3
>> test_functools leaked [0, 0, 3] memory blocks, sum=3
>> test_importlib leaked [5, 5, 5] references, sum=15
>> test_pkgutil leaked [1, 1, 1] references, sum=3
>> test_site leaked [0, 0, 2] references, sum=2
>> test_site leaked [0, 0, 2] memory blocks, sum=2
>> test_zipimport leaked [47, 47, 47] references, sum=141
>> test_zipimport_support leaked [2, 2, 2] references, sum=6
>>
>>
>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
>> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x']
>> ___
>> Python-checkins mailing list
>> python-check...@python.org
>> https://mail.python.org/mailman/listinfo/python-checkins
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (8391f99208f6): sum=181

[Eric Snow]

Fixed!  (test_site and test_functools are still leaking sporadically,
but it looks unrelated to the import.c leak).

-eric

On Thu, May 29, 2014 at 10:34 AM, Eric Snow  wrote:
> Good catch.  I'll look into it.
>
> -eric
>
> On Thu, May 29, 2014 at 8:49 AM, Brett Cannon  wrote:
>> I think the memory leak was caused by
>> http://hg.python.org/cpython/rev/7d20e30bd540 because
>> http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets the
>> 'res' variable and then overwrites it unconditionally w/o PY_DECREF
>> beforehand.
>>
>>
>> On Thu May 29 2014 at 4:02:17 AM,  wrote:
>>>
>>> results for 8391f99208f6 on branch "default"
>>> 
>>>
>>> test_collections leaked [0, 2, 4] references, sum=6
>>> test_collections leaked [0, 1, 2] memory blocks, sum=3
>>> test_functools leaked [0, 0, 3] memory blocks, sum=3
>>> test_importlib leaked [5, 5, 5] references, sum=15
>>> test_pkgutil leaked [1, 1, 1] references, sum=3
>>> test_site leaked [0, 0, 2] references, sum=2
>>> test_site leaked [0, 0, 2] memory blocks, sum=2
>>> test_zipimport leaked [47, 47, 47] references, sum=141
>>> test_zipimport_support leaked [2, 2, 2] references, sum=6
>>>
>>>
>>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
>>> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x']
>>> ___
>>> Python-checkins mailing list
>>> python-check...@python.org
>>> https://mail.python.org/mailman/listinfo/python-checkins
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Josiah Carlson]

Pinging this thread 2 months later with a progress/status update.

To those that have reviewed, commented, helped, or otherwise pushed this
along, which includes (but is not limited to) Richard Oudkerk, eryksun,
Giampaolo Rodola, thank you.


The short version:
As far as I can tell, the patch is ready:
http://bugs.python.org/issue1191964

What is available:
There are docs, tests, and obviously the functionality. Some code was moved
from asyncio/windows_utils.py (which has a separate issue here:
https://code.google.com/p/tulip/issues/detail?id=170). The API was changed
slightly from what was proposed by Guido:

sent = Popen.write_nonblocking(input, timeout=0)
data = Popen.read_nonblocking(bufsize=4096)
data = Popen.read_stderr_nonblocking(bufsize=4096)

As a bonus feature, Windows communicate() calls no longer spawn worker
threads, and instead use overlapped IO.


I'm bringing this back up to python-dev to offer a slightly wider audience
for commentary/concerns, and hopefully to get a stamp of approval that it
is ready.

Thank you,
 - Josiah




On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson 
wrote:

> I've got a patch with partial tests and documentation that I'm holding off
> on upload because I believe there should be a brief discussion.
>
> Long story short, Windows needs a thread to handle writing in a
> non-blocking fashion, regardless of the use of asyncio or plain subprocess.
>
> If you'd like to continue following this issue and participate in the
> discussion, I'll see you over on http://bugs.python.org/issue1191964 .
>
>  - Josiah
>
>
>
> On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson  > wrote:
>
>>
>> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum 
>> wrote:
>>
>>> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson <
>>> josiah.carl...@gmail.com> wrote:
>>>

 If it makes you feel any better, I spent an hour this morning building
 a 2-function API for Linux and Windows, both tested, not using ctypes, and
 not even using any part of asyncio (the Windows bits are in msvcrt and
 _winapi). It works in Python 3.3+. You can see it here:
 http://pastebin.com/0LpyQtU5

>>>
>>> Seeing this makes *me* feel better. I think it's reasonable to add (some
>>> variant) of that to the subprocess module in Python 3.5. It also belongs in
>>> the Activestate cookbook. And no, the asyncio module hasn't made it
>>> obsolete.
>>>
>>
>> Cool.
>>
>>  Josiah, you sound upset about the whole thing -- to the point of
>>> writing unintelligible sentences and passive-aggressive digs at everyone
>>> reading this list. I'm sorry that something happened that led you feel that
>>> way (if you indeed feel upset or frustrated) but I'm glad that you wrote
>>> that code snippet -- it is completely clear what you want and why you want
>>> it, and also what should happen next (a few rounds of code review on the
>>> tracker).
>>>
>>
>> I'm not always a prat. Something about python-dev brings out parts of me
>> that I thought I had discarded from my personality years ago. Toss in a bit
>> of needing to re-explain ideas that I've been trying to explain for almost
>> 9 years? Frustration + formerly discarded personality traits = uck. That's
>> pretty much why I won't be rejoining the party here regularly, you are all
>> better off without me commenting on 95% of threads like I used to.
>>
>> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I
>> was when I spend time on this list. That's *my* issue, not yours. That I
>> spent any time redirecting my frustration towards you is BS, and if I could
>> take back the email I sent just before getting Guido's, I would.
>>
>> I would advise everyone to write it off as the ramblings of a
>> surprisingly young, angry old man. Or call me an a-hole. Both are pretty
>> accurate. :)
>>
>> But that PEP? It's just a terrible PEP. It doesn't contain a single line
>>> of example code. It doesn't specify the proposed interface, it just
>>> describes in way too many sentences how it should work, and contains a
>>> whole lot of references to various rants from which the reader is
>>> apparently meant to become enlightened. I don't know which of the three
>>> authors *really* wrote it, and I don't want to know -- I think the PEP is
>>> irrelevant to the proposed feature, which is of "put it in the bug tracker
>>> and work from there" category -- presumably the PEP was written based on
>>> the misunderstanding that having a PEP would make acceptance of the patch
>>> easier, or because during an earlier bikeshedding round someone said
>>> "please write a PEP" (someone always says that). I propose to scrap the PEP
>>> (set the status to Withdrawn) and just work on adding the methods to the
>>> subprocess module.
>>>
>>
>> I'm not going to argue. The first I read it was 2-3 days ago.
>>
>>  If it were me, I'd define three methods, with longer names to clarify
>>> what they do, e.g.
>>>
>>> proc.write_nonblocking(data)
>>> data = proc.read_nonblocking()
>>> data =

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Josiah Carlson]

And as I was writing the "thank you" to folks, I hit send too early. Also
thank you to Victor Stinner, Guido, Terry Reedy, and everyone else on this
thread :)

 - Josiah


On Thu, May 29, 2014 at 5:34 PM, Josiah Carlson 
wrote:

> Pinging this thread 2 months later with a progress/status update.
>
> To those that have reviewed, commented, helped, or otherwise pushed this
> along, which includes (but is not limited to) Richard Oudkerk, eryksun,
> Giampaolo Rodola, thank you.
>
>
> The short version:
> As far as I can tell, the patch is ready:
> http://bugs.python.org/issue1191964
>
> What is available:
> There are docs, tests, and obviously the functionality. Some code was
> moved from asyncio/windows_utils.py (which has a separate issue here:
> https://code.google.com/p/tulip/issues/detail?id=170). The API was
> changed slightly from what was proposed by Guido:
>
> sent = Popen.write_nonblocking(input, timeout=0)
> data = Popen.read_nonblocking(bufsize=4096)
> data = Popen.read_stderr_nonblocking(bufsize=4096)
>
> As a bonus feature, Windows communicate() calls no longer spawn worker
> threads, and instead use overlapped IO.
>
>
> I'm bringing this back up to python-dev to offer a slightly wider audience
> for commentary/concerns, and hopefully to get a stamp of approval that it
> is ready.
>
> Thank you,
>  - Josiah
>
>
>
>
> On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson  > wrote:
>
>> I've got a patch with partial tests and documentation that I'm holding
>> off on upload because I believe there should be a brief discussion.
>>
>> Long story short, Windows needs a thread to handle writing in a
>> non-blocking fashion, regardless of the use of asyncio or plain subprocess.
>>
>> If you'd like to continue following this issue and participate in the
>> discussion, I'll see you over on http://bugs.python.org/issue1191964 .
>>
>>  - Josiah
>>
>>
>>
>> On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson <
>> josiah.carl...@gmail.com> wrote:
>>
>>>
>>> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum 
>>> wrote:
>>>
 On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson <
 josiah.carl...@gmail.com> wrote:

>
> If it makes you feel any better, I spent an hour this morning building
> a 2-function API for Linux and Windows, both tested, not using ctypes, and
> not even using any part of asyncio (the Windows bits are in msvcrt and
> _winapi). It works in Python 3.3+. You can see it here:
> http://pastebin.com/0LpyQtU5
>

 Seeing this makes *me* feel better. I think it's reasonable to add
 (some variant) of that to the subprocess module in Python 3.5. It also
 belongs in the Activestate cookbook. And no, the asyncio module hasn't made
 it obsolete.

>>>
>>> Cool.
>>>
>>>  Josiah, you sound upset about the whole thing -- to the point of
 writing unintelligible sentences and passive-aggressive digs at everyone
 reading this list. I'm sorry that something happened that led you feel that
 way (if you indeed feel upset or frustrated) but I'm glad that you wrote
 that code snippet -- it is completely clear what you want and why you want
 it, and also what should happen next (a few rounds of code review on the
 tracker).

>>>
>>> I'm not always a prat. Something about python-dev brings out parts of me
>>> that I thought I had discarded from my personality years ago. Toss in a bit
>>> of needing to re-explain ideas that I've been trying to explain for almost
>>> 9 years? Frustration + formerly discarded personality traits = uck. That's
>>> pretty much why I won't be rejoining the party here regularly, you are all
>>> better off without me commenting on 95% of threads like I used to.
>>>
>>> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I
>>> was when I spend time on this list. That's *my* issue, not yours. That I
>>> spent any time redirecting my frustration towards you is BS, and if I could
>>> take back the email I sent just before getting Guido's, I would.
>>>
>>> I would advise everyone to write it off as the ramblings of a
>>> surprisingly young, angry old man. Or call me an a-hole. Both are pretty
>>> accurate. :)
>>>
>>> But that PEP? It's just a terrible PEP. It doesn't contain a single line
 of example code. It doesn't specify the proposed interface, it just
 describes in way too many sentences how it should work, and contains a
 whole lot of references to various rants from which the reader is
 apparently meant to become enlightened. I don't know which of the three
 authors *really* wrote it, and I don't want to know -- I think the PEP is
 irrelevant to the proposed feature, which is of "put it in the bug tracker
 and work from there" category -- presumably the PEP was written based on
 the misunderstanding that having a PEP would make acceptance of the patch
 easier, or because during an earlier bikeshedding round someone said
 "please write a PEP" (someone always