Re: [Python-Dev] PEP 554 v4 (new interpreters module)

[Nick Coghlan]

On 6 December 2017 at 12:51, Eric Snow  wrote:
> Hi all,
>
> I've finally updated PEP 554.  Feedback would be most welcome.  The
> PEP is in a pretty good place now and I hope to we're close to a
> decision to accept it. :)

Nice updates! I like this version.

> In addition to resolving the open questions, I've also made the
> following changes to the PEP:
>
> * put an API summary at the top and moved the full API description down
> * add the "is_shareable()" function to indicate if an object can be shared
> * added None as a shareable object
>
> Regarding the open questions:
>
>  * "Leaking exceptions across interpreters"
>
> I chose to go with an approach that effectively creates a
> traceback.TracebackException proxy of the original exception, wraps
> that in a RuntimeError, and raises that in the calling interpreter.
> Raising an exception that safely preserves the original exception and
> traceback seems like the most intuitive behavior (to me, as a user).
> The only alternative that made sense is to fully duplicate the
> exception and traceback (minus stack frames) in the calling
> interpreter, which is probably overkill and likely to be confusing.

My one suggestion here would be to consider a dedicated exception type
like "interpreters.SubinterpreterError", rather than re-using
RuntimeError directly. That way you can put the extracted traceback on
a named attribute, and retain the option of potentially adding
subinterpreter awareness to the traceback module in the future.

>  * "Initial support for buffers in channels"
>
> I chose to add a "SendChannel.send_buffer(obj)" method for this.
> Supporting buffer objects from the beginning makes sense, opening good
> experimentation opportunities for a valuable set of users.  Supporting
> buffer objects separately and explicitly helps set clear expectations
> for users.  I decided not to go with a separate class (e.g.
> MemChannel) as it didn't seem like there's enough difference to
> warrant keeping them strictly separate.
>
> FWIW, I'm still strongly in favor of support for passing (copies of)
> bytes objects via channels.  Passing objects to SendChannel.send() is
> obvious.  Limiting it, for now, to bytes (and None) helps us avoid
> tying ourselves strongly to any particular implementation (it seems
> like all the reservations were relative to the implementation).  So I
> do not see a reason to wait.

Aye, the split sending API with a merged receive API works for me.

>  * "Pass channels explicitly to run()?"
>
> I've applied the suggested solution (make "channels" an explicit
> keyword argument).

Cool.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Nick Coghlan]

On 6 December 2017 at 16:18, Glenn Linderman  wrote:
> "b" mostly matters on Windows, correct? And Windows doesn't use C or POSIX
> locale, correct? And if these are correct, then is this an issue? And if so,
> why?

In Python 3, "b" matters everywhere, since it controls whether the
stream gets wrapped in TextIOWrapper or not.

It's only in Python 2 that the distinction is Windows-specific (where
it controls how "\r\n" sequences get handled).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Terry Reedy]

On 12/5/2017 10:25 AM, Mariatta Wijaya wrote:


* Time to merge a PR: 3 days in average, good!


Slide said 2.98 days, another said 4.4% by developers.

Regarding the average time to merge PR, I'm interested to know the 
average time to merge for PRs not made by Python Core Devs.


Trivially different: assume 0 days for all dev PRs, then average would 
be 2.96 / .956 = 3.12.


But any average that includes backports, which I suspect the above does, 
is skewed way down because backports are mostly merged soon after tests 
complete.  So I think 6 days average may be more realistic for master 
branch (3.7) PRs.


The average may be skewed down even more because some of the PRs that 
have been open a month or more will eventually be merged.


On the other hand, the mean, certainly by itself, is the *wrong* 
statistic for this data.  The argument for the value of using means 
depends on the data having a distribution that is at least roughly 
gaussian (misleadingly called 'normal').  The waiting times for merging 
appear to be negative exponential (slide 75).  Long waiting times have 
an oversized influence on the mean.


 So one should either calculate and report the median time or possibly 
the mean log(wait), converted back to waiting time.  In other words, 
exp(mean(map(log, waits))).  For the latter, one should probably start 
the clock after the initial CI tests finish, which is at least 1/2 hour.


--
Terry Jan Reedy (retired statistician)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v4 (new interpreters module)

[Petr Viktorin]

On 12/06/2017 03:51 AM, Eric Snow wrote:

Hi all,

I've finally updated PEP 554.  Feedback would be most welcome.  The
PEP is in a pretty good place now and I hope to we're close to a
decision to accept it. :)


[...]

C-extension opt-in/opt-out
--

By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily
add a mechanism by which C-extension modules could opt out of support
for subinterpreters.  Then the import machinery, when operating in
a subinterpreter, would need to check the module for support.  It would
raise an ImportError if unsupported. >
Alternately we could support opting in to subinterpreter support.
However, that would probably exclude many more modules (unnecessarily)
than the opt-out approach.


Currently it's already opt-in, as modules that use PyModuleDef are 
expected to support subinterpreters:

https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading

[...]

.. [global-atexit]
https://bugs.python.org/issue6531


Oh dear; there's now also https://bugs.python.org/issue31901
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Glenn Linderman]

On 12/5/2017 8:07 PM, INADA Naoki wrote:

Oh, revised version is really short!

And I have one worrying point.
With UTF-8 mode, open()'s default encoding/error handler is
UTF-8/surrogateescape.

Containers are really growing.  PyCharm supports Docker and many new Python
developers use Docker instead of installing Python directly on their system,
especially on Windows.

And opening binary file without "b" option is very common mistake of new
developers.  If default error handler is surrogateescape, they lose a chance
to notice their bug.


"b" mostly matters on Windows, correct? And Windows doesn't use C or 
POSIX locale, correct? And if these are correct, then is this an issue? 
And if so, why?



On the other hand, it helps some use cases when user want byte-transparent
behavior, without modifying code to use "surrogateescape" explicitly.

Which is more important scenario?  Anyone has opinion about it?
Are there any rationals and use cases I missing?

Regards,

INADA Naoki  


On Wed, Dec 6, 2017 at 12:17 PM, INADA Naoki  wrote:

I'm sorry about my laziness.
I've very busy these months, but I'm back to OSS world from today.

While I should review carefully again, I think I'm close to accept PEP 540.

* PEP 540 really helps containers and old Linux machines PEP 538 doesn't work.
   And containers is really important for these days.  Many new
Pythonistas who is
   not Linux experts start using containers.

* In recent years, UTF-8 fixed many mojibakes.  Now UnicodeError is
more usability
   problem for many Python users.  So I agree opt-out UTF-8 mode is
better than opt-in
   on POSIX locale.

I don't have enough time to read all mails in ML archive.
So if someone have opposite opinion, please remind me by this weekend.

Regards,

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Nick Coghlan]

On 6 December 2017 at 15:59, Chris Angelico  wrote:
> On Wed, Dec 6, 2017 at 4:46 PM, Nick Coghlan  wrote:
>> Something I've just noticed that needs to be clarified: on Linux, "C"
>> locale and "POSIX" locale are aliases, but this isn't true in general
>> (e.g. it's not the case on *BSD systems, including Mac OS X).
>
> For those of us with little to no BSD/MacOS experience, can you give a
> quick run-down of the differences between "C" and "POSIX"?

The one that's relevant to default locale detection is just the string
that "setlocale(LC_CTYPE, NULL)" returns.

On Linux (or, more accurately, with glibc), after setting
"LC_CTYPE=POSIX", that call still returns "C" (since the "POSIX"
locale is defined as an alias for the "C" locale).

By contrast, on *BSD, it will return "POSIX" (since "POSIX" is
actually a distinct locale there).

Beyond that, I don't know what the actual functional differences are.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Chris Angelico]

On Wed, Dec 6, 2017 at 4:46 PM, Nick Coghlan  wrote:
> Something I've just noticed that needs to be clarified: on Linux, "C"
> locale and "POSIX" locale are aliases, but this isn't true in general
> (e.g. it's not the case on *BSD systems, including Mac OS X).

For those of us with little to no BSD/MacOS experience, can you give a
quick run-down of the differences between "C" and "POSIX"?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Nick Coghlan]

Something I've just noticed that needs to be clarified: on Linux, "C"
locale and "POSIX" locale are aliases, but this isn't true in general
(e.g. it's not the case on *BSD systems, including Mac OS X).

To handle that in PEP 538, I made it clear that everything is keyed
specifically off the "C" locale, since that's what you actually get by
default.

So if PEP 540 is going to implicitly trigger switching encodings, it
needs to specify whether it's going to look for the C locale or the
POSIX locale (I'd suggest C locale, since that's the actual default
that causes problems).

The precedence relationship with locale coercion also needs to be
spelled out: successful locale coercion should skip implicitly
enabling UTF-8 mode (for opt-in UTF-8 mode, we'd still try to coerce
the locale setting as appropriate, so extensions modules are more
likely to behave themselves).

On 6 December 2017 at 14:07, INADA Naoki  wrote:
> Oh, revised version is really short!
>
> And I have one worrying point.
> With UTF-8 mode, open()'s default encoding/error handler is
> UTF-8/surrogateescape.
>
> Containers are really growing.  PyCharm supports Docker and many new Python
> developers use Docker instead of installing Python directly on their system,
> especially on Windows.
>
> And opening binary file without "b" option is very common mistake of new
> developers.  If default error handler is surrogateescape, they lose a chance
> to notice their bug.
>
> On the other hand, it helps some use cases when user want byte-transparent
> behavior, without modifying code to use "surrogateescape" explicitly.
>
> Which is more important scenario?  Anyone has opinion about it?
> Are there any rationals and use cases I missing?

For platforms that offer a C.UTF-8 locale, I'd like "LC_CTYPE=C.UTF-8
python" and "PYTHONCOERCECLOCALE=0 LC_CTYPE=C PYTHONUTF8=1" to be
equivalent (aside from the known limitation that extension modules may
not do the right thing in the latter case).

For the locale coercion case, the default error handler for `open`
remains as "strict", which means I'd be in favour of keeping it as
"strict" by default in UTF-8 mode as well. That would flip the toggle
in the PEP: "strict UTF-8" would be the default selection for
"PYTHONUTF8=1, and you'd choose the more relaxed option via
"PYTHONUTF8=permissive".

That way, the combination of PEPs 538 and 540 would give us the
following situation in the C locale:

1. Our preferred approach is to coerce LC_CTYPE in the C locale to a
UTF-8 based equivalent
2. Only if that fails (e.g. as it will on CentOS 7) do we resort to
implicitly enabling CPython's internal UTF-8 mode (which should behave
like C.UTF-8, *except* for the fact extension modules won't respect
it)

That way, the ideal outcome is that a UTF-8 based locale exists, and
we use it automatically when needed. UTF-8 mode than lets us cope with
older platforms where neither C.UTF-8 nor an equivalent exists.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Nick Coghlan]

On 6 December 2017 at 14:34, Nick Coghlan  wrote:
> That said, I go agree we could offer easier to use APIs to app
> developers that just want to hide warnings from their users, so I've
> filed https://bugs.python.org/issue32229 to propose a straightforward
> "warnings.hide_warnings()" API that encapsulates things like checking
> for a non-empty sys.warnoptions list.

I've updated the "Limitations" section of the PEP to mention that
separate proposal:
https://github.com/python/peps/commit/6e93c8d2e6ad698834578d4077b92a8fc84a70f5

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Nick Coghlan]

On 6 December 2017 at 06:43, Victor Stinner  wrote:
> At the end, I'm not sure that the PEP 565 is really needed or would help 
> anyone.

Folks, I'd really appreciate it if you could comment on the merits of
the PEP without implicitly claiming that I don't exist, and that Linux
system administrators don't exist :)

Right now, Linux distro Python library upgrades (including CPython
updates to a new feature release) may result in *hard compatibility
breaks*, as previously deprecated features disappear without warning.

With PEP 565, ad hoc personal scripts will at least get a
DeprecationWarning when APIs they're using are at risk of disappearing
as a result of a distro package or version update. Right now, they
don't get a warning at all.

There is *no way* for any opt in flag to help these users. Now, PEP
565 being rejected wouldn't be the end of the world on that front - we
can apply comparable changes as a downstream patch. However, these
problems aren't unique to Linux, and they aren't unique to any
specific distro: they apply whenever folks are using Python for
personal workflow automation, rather than for redistributable
applications and libraries. So it makes more sense to me to do this
upstream, rather than having the default warnings handling be a
redistributor dependent behaviour.

That said, I go agree we could offer easier to use APIs to app
developers that just want to hide warnings from their users, so I've
filed https://bugs.python.org/issue32229 to propose a straightforward
"warnings.hide_warnings()" API that encapsulates things like checking
for a non-empty sys.warnoptions list.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[INADA Naoki]

Oh, revised version is really short!

And I have one worrying point.
With UTF-8 mode, open()'s default encoding/error handler is
UTF-8/surrogateescape.

Containers are really growing.  PyCharm supports Docker and many new Python
developers use Docker instead of installing Python directly on their system,
especially on Windows.

And opening binary file without "b" option is very common mistake of new
developers.  If default error handler is surrogateescape, they lose a chance
to notice their bug.

On the other hand, it helps some use cases when user want byte-transparent
behavior, without modifying code to use "surrogateescape" explicitly.

Which is more important scenario?  Anyone has opinion about it?
Are there any rationals and use cases I missing?

Regards,

INADA Naoki  


On Wed, Dec 6, 2017 at 12:17 PM, INADA Naoki  wrote:
> I'm sorry about my laziness.
> I've very busy these months, but I'm back to OSS world from today.
>
> While I should review carefully again, I think I'm close to accept PEP 540.
>
> * PEP 540 really helps containers and old Linux machines PEP 538 doesn't work.
>   And containers is really important for these days.  Many new
> Pythonistas who is
>   not Linux experts start using containers.
>
> * In recent years, UTF-8 fixed many mojibakes.  Now UnicodeError is
> more usability
>   problem for many Python users.  So I agree opt-out UTF-8 mode is
> better than opt-in
>   on POSIX locale.
>
> I don't have enough time to read all mails in ML archive.
> So if someone have opposite opinion, please remind me by this weekend.
>
> Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Nick Coghlan]

On 6 December 2017 at 05:11, Barry Warsaw  wrote:
> On Dec 5, 2017, at 13:24, Guido van Rossum  wrote:
>
>> But the whole point of the PEP is that it only warns about deprecations in 
>> code over which the user has control -- likely __main__ is their own code, 
>> and they *can* handle it.
>
> I’m not so sure how true that is.  I have no sense of the relative popularity 
> of hand crafted dunder-mains vs entry point crafted ones.  I know that in my 
> own applications, I tend to use the latter (although pkg_resources 
> performance issues bum me out).  But then you have applications like pex that 
> use fairly complex hand crafted dunder-mains in their zip files.  In either 
> case I don’t want consumers of my applications to have to worry about 
> DeprecationWarnings, since *they* really can’t do anything about them.

For something like pex, it would likely be appropriate to add the
following to __main__.py:

import sys, warnings
if not sys.warnoptions:
   warnings.simplefilter("ignore")

We don't currently provide that snippet anywhere in the docs, so the
PEP suggests we add it:
https://www.python.org/dev/peps/pep-0565/#other-documentation-updates

> All that to say I really don’t know what the right thing to do here is.  All 
> of our fiddling with the reporting of DeprecationWarnings, not to mention 
> PendingDeprecationWarnings and FutureWarnings feels like experimental shots 
> in the dark, and I suspect we won’t really know if PEP 565 will be helpful, 
> harmful, or neutral until it’s out in the wild for a while.  I suspect either 
> that what we’re trying to accomplish really can’t be done, or that we really 
> don’t have a good understanding of the problem and we’re just chipping away 
> at the edges.

I'm entirely comfortable with telling app developers "Include this 3
line snippet if you don't want your users to see any warnings by
default".

Historically, we've never provided good documentation on how to do
that, so apps have tended to rely on the default filters, and we've
then had ongoing arguments about who wins and who loses in deciding
what the defaults should be.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[INADA Naoki]

I'm sorry about my laziness.
I've very busy these months, but I'm back to OSS world from today.

While I should review carefully again, I think I'm close to accept PEP 540.

* PEP 540 really helps containers and old Linux machines PEP 538 doesn't work.
  And containers is really important for these days.  Many new
Pythonistas who is
  not Linux experts start using containers.

* In recent years, UTF-8 fixed many mojibakes.  Now UnicodeError is
more usability
  problem for many Python users.  So I agree opt-out UTF-8 mode is
better than opt-in
  on POSIX locale.

I don't have enough time to read all mails in ML archive.
So if someone have opposite opinion, please remind me by this weekend.

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 554 v4 (new interpreters module)

[Eric Snow]

Hi all,

I've finally updated PEP 554.  Feedback would be most welcome.  The
PEP is in a pretty good place now and I hope to we're close to a
decision to accept it. :)

In addition to resolving the open questions, I've also made the
following changes to the PEP:

* put an API summary at the top and moved the full API description down
* add the "is_shareable()" function to indicate if an object can be shared
* added None as a shareable object

Regarding the open questions:

 * "Leaking exceptions across interpreters"

I chose to go with an approach that effectively creates a
traceback.TracebackException proxy of the original exception, wraps
that in a RuntimeError, and raises that in the calling interpreter.
Raising an exception that safely preserves the original exception and
traceback seems like the most intuitive behavior (to me, as a user).
The only alternative that made sense is to fully duplicate the
exception and traceback (minus stack frames) in the calling
interpreter, which is probably overkill and likely to be confusing.

 * "Initial support for buffers in channels"

I chose to add a "SendChannel.send_buffer(obj)" method for this.
Supporting buffer objects from the beginning makes sense, opening good
experimentation opportunities for a valuable set of users.  Supporting
buffer objects separately and explicitly helps set clear expectations
for users.  I decided not to go with a separate class (e.g.
MemChannel) as it didn't seem like there's enough difference to
warrant keeping them strictly separate.

FWIW, I'm still strongly in favor of support for passing (copies of)
bytes objects via channels.  Passing objects to SendChannel.send() is
obvious.  Limiting it, for now, to bytes (and None) helps us avoid
tying ourselves strongly to any particular implementation (it seems
like all the reservations were relative to the implementation).  So I
do not see a reason to wait.

 * "Pass channels explicitly to run()?"

I've applied the suggested solution (make "channels" an explicit
keyword argument).

-eric


I've include the latest full text
(https://www.python.org/dev/peps/pep-0554/) below:

+

PEP: 554
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.7
Post-History: 07-Sep-2017, 08-Sep-2017, 13-Sep-2017, 05-Dec-2017


Abstract


CPython has supported multiple interpreters in the same process (AKA
"subinterpreters") since version 1.5.  The feature has been available
via the C-API. [c-api]_ Subinterpreters operate in
`relative isolation from one another `_, which
provides the basis for an
`alternative concurrency model `_.

This proposal introduces the stdlib ``interpreters`` module.  The module
will be `provisional `_.  It exposes the basic
functionality of subinterpreters already provided by the C-API, along
with new functionality for sharing data between interpreters.


Proposal


The ``interpreters`` module will be added to the stdlib.  It will
provide a high-level interface to subinterpreters and wrap a new
low-level ``_interpreters`` (in the same was as the ``threading``
module).  See the `Examples`_ section for concrete usage and use cases.

Along with exposing the existing (in CPython) subinterpreter support,
the module will also provide a mechanism for sharing data between
interpreters.  This mechanism centers around "channels", which are
similar to queues and pipes.

Note that *objects* are not shared between interpreters since they are
tied to the interpreter in which they were created.  Instead, the
objects' *data* is passed between interpreters.  See the `Shared data`_
section for more details about sharing between interpreters.

At first only the following types will be supported for sharing:

* None
* bytes
* PEP 3118 buffer objects (via ``send_buffer()``)

Support for other basic types (e.g. int, Ellipsis) will be added later.

API summary for interpreters module
---

Here is a summary of the API for the ``interpreters`` module.  For a
more in-depth explanation of the proposed classes and functions, see
the `"interpreters" Module API`_ section below.

For creating and using interpreters:

+--+--+
| signature| description  |
++=+==+
| list_all() -> [Intepreter]   | Get all existing interpreters.   |
+--+--+
| get_current() -> Interpreter | Get the currently running interpreter.   |
+--+--+
| create() -> Interpreter  | Initialize a new (idle) Python interpreter.  |

[Python-Dev] [RELEASE] Python 3.6.4rc1 and 3.7.0a3 now available for testing

[Ned Deily]

Announcing the immediate availability of Python 3.6.4 release candidate 1
and of Python 3.7.0 alpha 3!

Python 3.6.4rc1 is the first release candidate for Python 3.6.4, the next
maintenance release of Python 3.6.  While 3.6.4rc1 is a preview release and,
thus, not intended for production environments, we encourage you to explore
it and provide feedback via the Python bug tracker (https://bugs.python.org).
3.6.4 is planned for final release on 2017-12-18 with the next maintenance
release expected to follow in about 3 months.  You can find Python 3.6.4rc1
and more information here:
https://www.python.org/downloads/release/python-364rc1/

Python 3.7.0a3 is the third of four planned alpha releases of Python 3.7,
the next feature release of Python.  During the alpha phase, Python 3.7
remains under heavy development: additional features will be added
and existing features may be modified or deleted.  Please keep in mind
that this is a preview release and its use is not recommended for
production environments.  The next preview release, 3.7.0a4, is planned
for 2018-01-08. You can find Python 3.7.0a3 and more information here:
https://www.python.org/downloads/release/python-370a3/

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Nick Coghlan]

On 6 December 2017 at 11:01, Victor Stinner  wrote:
>> Annex: Differences between the PEP 538 and the PEP 540
>> ==
>>
>> The PEP 538 uses the "C.UTF-8" locale which is quite new and only
>> supported by a few Linux distributions; this locale is not currently
>> supported by FreeBSD or macOS for example. This PEP 540 supports all
>> operating systems.
>>
>> The PEP 538 only changes the behaviour for the POSIX locale. While the
>> new UTF-8 mode of this PEP is only enabled by the POSIX locale, it can
>> be enabled manually for any other locale.
>>
>> The PEP 538 is implemented with ``setlocale(LC_CTYPE, "C.UTF-8")``: any
>> non-Python code running in the process is impacted by this change.  This
>> PEP is implemented in Python internals and ignores the locale:
>> non-Python running in the same process is not aware of the "Python UTF-8
>> mode".

I submitted a PR to reword this part: https://github.com/python/peps/pull/493

> The main advantage of the PEP 538 ùover* the PEP 540 is that, for the
> POSIX locale, non-Python code running in the same process gets the
> UTF-8 encoding.
>
> To be honest, I'm not sure that there is a lot of code in the wild
> which uses "text" types like the C type wchar_t* and rely on the
> locale encoding. Almost all C library handle data as bytes using the
> char* type, like filenames and environment variables.

At the very least, GNU readline breaks if you don't change the locale
setting: 
https://www.python.org/dev/peps/pep-0538/#considering-locale-coercion-independently-of-utf-8-mode

Given that we found an example of this directly in the standard
library, I assume that there are plenty more in third party extension
modules (especially once we take C++ extensions into account, not just
C ones).

> First I understood that the PEP 538 changed the locale encoding using
> an environment variable. But no, it's implemented with
> setlocale(LC_CTYPE, "C.UTF-8") which only impacts the current process
> and is not inherited by child processes. So I'm not sure anymore that
> PEP 538 and PEP 540 are really complementary.

It sets the LC_CTYPE environment variable as well:
https://www.python.org/dev/peps/pep-0538/#explicitly-setting-lc-ctype-for-utf-8-locale-coercion

The relevant code is in _coerce_default_locale_settings (currently at
https://github.com/python/cpython/blob/master/Python/pylifecycle.c#L448)

> I'm not sure how PyGTK interacts with the PEP 538 for example. Does it
> use UTF-8 with the POSIX locale?

Desktop environments aim not to get into this situation in the first
place by ensuring they're using a more appropriate locale :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Victor Stinner]

> Annex: Differences between the PEP 538 and the PEP 540
> ==
>
> The PEP 538 uses the "C.UTF-8" locale which is quite new and only
> supported by a few Linux distributions; this locale is not currently
> supported by FreeBSD or macOS for example. This PEP 540 supports all
> operating systems.
>
> The PEP 538 only changes the behaviour for the POSIX locale. While the
> new UTF-8 mode of this PEP is only enabled by the POSIX locale, it can
> be enabled manually for any other locale.
>
> The PEP 538 is implemented with ``setlocale(LC_CTYPE, "C.UTF-8")``: any
> non-Python code running in the process is impacted by this change.  This
> PEP is implemented in Python internals and ignores the locale:
> non-Python running in the same process is not aware of the "Python UTF-8
> mode".

The main advantage of the PEP 538 ùover* the PEP 540 is that, for the
POSIX locale, non-Python code running in the same process gets the
UTF-8 encoding.

To be honest, I'm not sure that there is a lot of code in the wild
which uses "text" types like the C type wchar_t* and rely on the
locale encoding. Almost all C library handle data as bytes using the
char* type, like filenames and environment variables.

First I understood that the PEP 538 changed the locale encoding using
an environment variable. But no, it's implemented with
setlocale(LC_CTYPE, "C.UTF-8") which only impacts the current process
and is not inherited by child processes. So I'm not sure anymore that
PEP 538 and PEP 540 are really complementary.

I'm not sure how PyGTK interacts with the PEP 538 for example. Does it
use UTF-8 with the POSIX locale?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Victor Stinner]

Hi,

I knew that I had to rewrite my PEP 540, but I was too lazy. Since
Guido explicitly requested a shorter PEP, here you have!

https://www.python.org/dev/peps/pep-0540/

Trust me, it's the same PEP, but focused on the most important
information and with a shorter rationale ;-)

Full text below.

Victor


PEP: 540
Title: Add a new UTF-8 mode
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner 
BDFL-Delegate: INADA Naoki
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 5-January-2016
Python-Version: 3.7


Abstract


Add a new UTF-8 mode to ignore the locale and use the UTF-8 encoding
with the ``surrogateescape`` error handler. This mode is enabled by
default in the POSIX locale, but otherwise disabled by default.

Add also a "strict" UTF-8 mode which uses the ``strict`` error handler,
instead of ``surrogateescape``, with the UTF-8 encoding.

The new ``-X utf8`` command line option and ``PYTHONUTF8`` environment
variable are added to control the UTF-8 mode.


Rationale
=

Locale encoding and UTF-8
-

Python 3.6 uses the locale encoding for filenames, environment
variables, standard streams, etc. The locale encoding is inherited from
the locale; the encoding and the locale are tightly coupled.

Many users inherit the ASCII encoding from the POSIX locale, aka the "C"
locale, but are unable change the locale for different reasons. This
encoding is very limited in term of Unicode support: any non-ASCII
character is likely to cause troubles. For example, the Alpine Linux
distribution became popular thanks to Docker containers, but it uses the
POSIX locale by default.

It is not easy to get the expected locale. Locales don't get the exact
same name on all Linux distributions, FreeBSD, macOS, etc. Some
locales, like the recent ``C.UTF-8`` locale, are only supported by a few
platforms. For example, a SSH connection can use a different encoding
than the filesystem or terminal encoding of the local host.

On the other side, Python 3.6 is already using UTF-8 by default on
macOS, Android and Windows (PEP 529) for most functions, except of
``open()``. UTF-8 is also the default encoding of Python scripts, XML
and JSON file formats. The Go programming language uses UTF-8 for
strings.

When all data are stored as UTF-8 but the locale is often misconfigured,
an obvious solution is to ignore the locale and use UTF-8.

Passthough undecodable bytes: surrogateescape
-

Using UTF-8 is nice, until you read the first file encoded to a
different encoding. When using the ``strict`` error handler, which is
the default, Python 3 raises a ``UnicodeDecodeError`` on the first
undecodable byte.

Unix command line tools like ``cat`` or ``grep`` and most Python 2
applications simply do not have this class of bugs: they don't decode
data, but process data as a raw bytes sequence.

Python 3 already has a solution to behave like Unix tools and Python 2:
the ``surrogateescape`` error handler (:pep:`383`). It allows to process
data "as bytes" but uses Unicode in practice (undecodable bytes are
stored as surrogate characters).

For an application written as a Unix "pipe" tool like ``grep``, taking
input on stdin and writing output to stdout, ``surrogateescape`` allows
to "passthrough" undecodable bytes.

The UTF-8 encoding used with the ``surrogateescape`` error handler is a
compromise between correctness and usability.

Strict UTF-8 for correctness


When correctness matters more than usability, the ``strict`` error
handler is preferred over ``surrogateescape`` to raise an encoding error
at the first undecodable byte or unencodable character.

No change by default for best backward compatibility


While UTF-8 is perfect in most cases, sometimes the locale encoding is
actually the best encoding.

This PEP changes the behaviour for the POSIX locale since this locale
usually gives the ASCII encoding, whereas UTF-8 is a much better choice.
It does not change the behaviour for other locales to prevent any risk
or regression.

As users are responsible to enable explicitly the new UTF-8 mode, they
are responsible for any potential mojibake issues caused by this mode.


Proposal


Add a new UTF-8 mode to ignore the locale and use the UTF-8 encoding
with the ``surrogateescape`` error handler. This mode is enabled by
default in the POSIX locale, but otherwise disabled by default.

Add also a "strict" UTF-8 mode which uses the ``strict`` error handler,
instead of ``surrogateescape``, with the UTF-8 encoding.

The new ``-X utf8`` command line option and ``PYTHONUTF8`` environment
variable are added to control the UTF-8 mode:

* The UTF-8 mode is enabled by ``-X utf8`` or ``PYTHONUTF8=1``
* The Strict UTF-8 mode is configured by ``-X utf8=strict`` or
  ``PYTHONUTF8=strict``

The POSIX locale enables the UTF-8 mode. In this case, 

Re: [Python-Dev] Zero-width matching in regexes

[MRAB]

On 2017-12-05 20:26, Terry Reedy wrote:

On 12/4/2017 6:21 PM, MRAB wrote:
I've finally come to a conclusion as to what the "correct" behaviour of 
zero-width matches should be: """always return the first match, but 
never a zero-width match that is joined to a previous zero-width match""".


Is this different from current re or regex?


Sometimes yes.

It's difficult to know how a zero-width match should be handled.

The normal way that, say, findall works is that it searches for a match 
and then continues from where it left off.


If at any point it matched an empty string, it would stall because the 
end of the match is also the start of the match.


How should that be handled?

The old buggy behaviour of the re module was to just advance by one 
character after a zero-width match, which can result in a character 
being skipped and going missing.


A solution is to prohibit a zero-width match that's joined to the 
previous match, but I'm not convinced that that's correct.


If it's about to return a zero-width match that's joined to a previous 
zero-width match, then backtrack and keep on looking for a match.


Example:

 >>> print([m.span() for m in re.finditer(r'|.', 'a')])
[(0, 0), (0, 1), (1, 1)]

re.findall, re.split and re.sub should work accordingly.

If re.finditer finds n matches, then re.split should return a list of 
n+1 strings and re.sub should make n replacements (excepting maxsplit, 
etc.).



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

[Victor Stinner]

Chris:
> I just took another look at 538 -- and yes, the relationship between the two
> is really unclear. In particular, with 538, why do we need 540? I honestly
> don't know.

The PEP 538 only impacts platforms which provide the C.UTF-8 locale or
a variant: only a few recent Linux distribution. I know Fedora, maybe
a few other have it? FreeBSD and macOS are completely ignored by the
PEP 538. The PEP 540 uses the UTF-8 encoding for the POSIX locale on
*all* platforms.

Moreover, the PEP 538 only concerns the POSIX locale (locale "C"),
whereas the PEP 540 is usable with any locale. For example, using the
"fr_FR.iso88591" locale, the encoding is Latin1. But if you enable the
UTF-8 mode with this locale, Python will use UTF-8.

The other difference is that the PEP 538 is implemented with
setlocale(LC_CTYPE, "C.UTF-8"), whereas the PEP 540 is implemented in
Python internals and ignores the locale. The PEP 540 scope is limited
to Python, non-Python running in the same process is not aware of the
"Python UTF-8 mode".

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

[Chris Barker]

On Tue, Dec 5, 2017 at 1:18 PM, Guido van Rossum  wrote:
>
>
> I am very worried about this long and rambling PEP,
>

FWIW, I read the PEP on the bus this morning on a phone, and while lng, I
didn't find it too rambling. And this topic has been very often discussed
in very long and rambling mailing list threads, etc. So I think a long (If
not rambling) PEP is in order.

This is a very important topic for Python -- the py2-3 transition got a LOT
of flack, to the point of people claiming that it was easier to learn a
whole new language than convert to py3 -- and THIS particular issue was a
big part of it:

The truth is that any system that does not use a clearly defined encoding
for filenames (and everything else) is broken, plain and simple. But the
other truth is (as talked about in the PEP) they some *nix systems are that
broken because C code that simply passed around char* still works fine. And
no matter how you slice it telling people that they need to fix their
broken system in order for your software to run is not a popular option.

When Python added surrogateescape to its Unicode implementation, the tools
were there to work with broken (OK, I'll be charitable: misconfigured)
systems. Now we just need some easier defaults.

OK, now I'm getting long and rambling

TL;DR -- The proposal in the PEP is an important step forward, and the
issue is fraught with enough history and controversy that a long PEP is
probably a good idea.

So the addition of a better summary of the specification up at the top, and
editing of the rest, and we could have a good PEP.

Too late for this release, but what can you do?


> The "Unicode just works" summary is more a wish than a proper summary of
> the PEP.
>

well, yeah.


> FWIW the relationship with PEP 538 is also pretty unclear. (Or maybe
> that's another case of the forest and the trees.) And that PEP (while
> already accepted) also comes across as rambling and vague, and I have no
> idea what it actually does. And it seems to mention PEP 540 quite a few
> times.
>

I just took another look at 538 -- and yes, the relationship between the
two is really unclear. In particular, with 538, why do we need 540? I
honestly don't know.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

[Victor Stinner]

2017-12-05 22:18 GMT+01:00 Guido van Rossum :
> So I guess PEP acceptance week is over. :-(

My bad, Barry wrote "PEP Acceptance Day", not week, on twitter ;-)
https://twitter.com/pumpichank/status/937770805076905990

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode

[Guido van Rossum]

I've been discussing this PEP offline with Victor, but he suggested we
should discuss it in public instead.

I am very worried about this long and rambling PEP, and I propose that it
not be accepted without a major rewrite to focus on clarity of the
specification. The "Unicode just works" summary is more a wish than a
proper summary of the PEP.

For others interested in reviewing this, the implementation link is hidden
in the long list of links; it is http://bugs.python.org/issue29240.


FWIW the relationship with PEP 538 is also pretty unclear. (Or maybe that's
another case of the forest and the trees.) And that PEP (while already
accepted) also comes across as rambling and vague, and I have no idea what
it actually does. And it seems to mention PEP 540 quite a few times.

So I guess PEP acceptance week is over. :-(

On Tue, Dec 5, 2017 at 7:52 AM, Victor Stinner 
wrote:

> Hi,
>
> Since it's the PEP Acceptance Week, I try my luck! Here is my very
> long PEP to propose a tiny change. The PEP is very long to explain the
> rationale and limitations.
>
> Inaccurate tl; dr with the UTF-8 mode, Unicode "just works" as expected.
>
> Reminder: INADA Naoki was nominated as the BDFL-Delegate.
>
> https://www.python.org/dev/peps/pep-0540/
>
> Full-text below.
>
> Victor
>
>
> PEP: 540
> Title: Add a new UTF-8 mode
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner ,
> Nick Coghlan 
> BDFL-Delegate: INADA Naoki
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 5-January-2016
> Python-Version: 3.7
>
>
> Abstract
> 
>
> Add a new UTF-8 mode, enabled by default in the POSIX locale, to ignore
> the locale and force the usage of the UTF-8 encoding for external
> operating system interfaces, including the standard IO streams.
>
> Essentially, the UTF-8 mode behaves as Python 2 and other C based
> applications on \*nix systems: it aims to process text as best it can,
> but it errs on the side of producing or propagating mojibake to
> subsequent components in a processing pipeline rather than requiring
> strictly valid encodings at every step in the process.
>
> The UTF-8 mode can be configured as strict to reduce the risk of
> producing or propagating mojibake.
>
> A new ``-X utf8`` command line option and ``PYTHONUTF8`` environment
> variable are added to explicitly control the UTF-8 mode (including
> turning it off entirely, even in the POSIX locale).
>
>
> Rationale
> =
>
> "It's not a bug, you must fix your locale" is not an acceptable answer
> --
>
> Since Python 3.0 was released in 2008, the usual answer to users getting
> Unicode errors is to ask developers to fix their code to handle Unicode
> properly. Most applications and Python modules were fixed, but users
> kept reporting Unicode errors regularly: see the long list of issues in
> the `Links`_ section below.
>
> In fact, a second class of bugs comes from a locale which is not properly
> configured. The usual answer to such a bug report is: "it is not a bug,
> you must fix your locale".
>
> Technically, the answer is correct, but from a practical point of view,
> the answer is not acceptable. In many cases, "fixing the issue" is a
> hard task. Moreover, sometimes, the usage of the POSIX locale is
> deliberate.
>
> A good example of a concrete issue are build systems which create a
> fresh environment for each build using a chroot, a container, a virtual
> machine or something else to get reproducible builds. Such a setup
> usually uses the POSIX locale.  To get 100% reproducible builds, the
> POSIX locale is a good choice: see the `Locales section of
> reproducible-builds.org
> `_.
>
> PEP 538 lists additional problems related to the use of Linux containers to
> run network services and command line applications.
>
> UNIX users don't expect Unicode errors, since the common command lines
> tools like ``cat``, ``grep`` or ``sed`` never fail with Unicode errors -
> they produce mostly-readable text instead.
>
> These users similarly expect that tools written in Python 3 (including
> those updated from Python 2), continue to tolerate locale
> misconfigurations and avoid bothering them with text encoding details.
> From their point of the view, the bug is not their locale but is
> obviously Python 3 ("Everything else works, including Python 2, so
> what's wrong with Python 3?").
>
> Since Python 2 handles data as bytes, similar to system utilities
> written in C and C++, it's rarer in Python 2 compared to Python 3 to get
> explicit Unicode errors. It also contributes significantly to why many
> affected users perceive Python 3 as the root cause of their Unicode
> errors.
>
> At the same time, the stricter text handling model was deliberately
> introduced into Python 3 to 

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Victor Stinner]

2017-12-05 21:21 GMT+01:00 Serhiy Storchaka :
> 05.12.17 22:10, Victor Stinner пише:
>>
>> Maybe we need something to declare the code that we own, to
>> enable warnings on them.
>
> Just compare __author__ with the name of the user running a script. ;-)

I was thinking as something like:

   enable_warnings_on('app')

which would enable warnings on app.py and app/* including
subdirectories (app/tools.py but also app/submodule/thing.py).

But the drawback of this idea compared to PEP 565 is that it requires
to modify each application, whereas the PEP 565 changes the default
behaviour.


Warnings are a complex beast.

If you think about ResourceWarning: the warning can be emited "in the
stdlib", whereas the file was opened in "your own code".
ResourceWarning are basically emited *anywhere* because of their weird
nature. It's often that these warnings are emited on a garbage
collection, and this can happen anytime, usually far from where the
resource was allocated. (Since Python 3.6, if you enable tracemalloc,
the ResourceWarning is now logged with the traceback where the
resource was allocated!!!)

-b and -bb options are global options to change the behaviour of
BytesWarning: it's not possible to only make BytesWarning strict "in
your own code".

(I'm not sure neither that all code uses properly the warnings API to
identify correctly the caller where the warning should be emitted.)

...

For all these reasons, I prefer to not try to *guess* what is the
"owned code", but simply always emit warnings everywhere :-) That's
why I suggest to use -W default option or PYTHONWARNINGS=default (or
the new -X dev option or PYTHONDEVMODE=1).

In my experience, it's fine to get warnings in third party code. I'm
lucky, I'm only working on open source softwares, so I can report and
even fix (!) warnings in the code that I don't own :-)

By the way, if you are annoyed by one very specific warning in
external code (which prevents you to fix other warnings "further" in
the code), you can ignore it using a filter. You can specify a module
name, and even a line number of a module.

   https://docs.python.org/dev/library/warnings.html#warnings.filterwarnings

At the end, I'm not sure that the PEP 565 is really needed or would help anyone.

Again, I understand that there is no perfect solution, and that PEP
565 is a compromise.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Zero-width matching in regexes

[Terry Reedy]

On 12/4/2017 6:21 PM, MRAB wrote:
I've finally come to a conclusion as to what the "correct" behaviour of 
zero-width matches should be: """always return the first match, but 
never a zero-width match that is joined to a previous zero-width match""".


Is this different from current re or regex?

If it's about to return a zero-width match that's joined to a previous 
zero-width match, then backtrack and keep on looking for a match.


Example:

 >>> print([m.span() for m in re.finditer(r'|.', 'a')])
[(0, 0), (0, 1), (1, 1)]

re.findall, re.split and re.sub should work accordingly.

If re.finditer finds n matches, then re.split should return a list of 
n+1 strings and re.sub should make n replacements (excepting maxsplit, 
etc.).



--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Serhiy Storchaka]

05.12.17 22:10, Victor Stinner пише:

Maybe we need something to declare the code that we own, to
enable warnings on them.

Just compare __author__ with the name of the user running a script. ;-)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Guido van Rossum]

On Tue, Dec 5, 2017 at 12:10 PM, Victor Stinner 
wrote:

> 2017-12-05 19:24 GMT+01:00 Guido van Rossum :
> >> I disagree that *users* of an application is supposed to "handle"
> >> deprecation warnings: report them to the developer, or even try to fix
> >> them. IHMO these warnings (hidden by default) were introduced for
> >> developers of the application.
> >
> > But the whole point of the PEP is that it only warns about deprecations
> in
> > code over which the user has control -- likely __main__ is their own
> code,
> > and they *can* handle it.
>
> IMHO the core of the PEP 565 is to propose a compromise to separate
> "own code" and "external code" (cannot be modified).
>
> I'm unhappy with this suboptimal compromise: "only __main__ is my own
> code". Maybe we need something to declare the code that we own, to
> enable warnings on them. Just a simple helper on top of
> warnings.filterwarnings()? Or maybe I'm already in the "the better is
> the enemy of the good" greay area :-)
>

This is a very good characterization of the dilemma. But IMO we don't have
a better proxy for "my own code" (it's tempting to try and define it based
on the current directory but this makes too many assumptions about
development styles).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Victor Stinner]

2017-12-05 19:24 GMT+01:00 Guido van Rossum :
>> I disagree that *users* of an application is supposed to "handle"
>> deprecation warnings: report them to the developer, or even try to fix
>> them. IHMO these warnings (hidden by default) were introduced for
>> developers of the application.
>
> But the whole point of the PEP is that it only warns about deprecations in
> code over which the user has control -- likely __main__ is their own code,
> and they *can* handle it.

IMHO the core of the PEP 565 is to propose a compromise to separate
"own code" and "external code" (cannot be modified).

I'm unhappy with this suboptimal compromise: "only __main__ is my own
code". Maybe we need something to declare the code that we own, to
enable warnings on them. Just a simple helper on top of
warnings.filterwarnings()? Or maybe I'm already in the "the better is
the enemy of the good" greay area :-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Guido van Rossum]

On Tue, Dec 5, 2017 at 11:11 AM, Barry Warsaw  wrote:

> On Dec 5, 2017, at 13:24, Guido van Rossum  wrote:
>
> > But the whole point of the PEP is that it only warns about deprecations
> in code over which the user has control -- likely __main__ is their own
> code, and they *can* handle it.
>
> I’m not so sure how true that is.  I have no sense of the relative
> popularity of hand crafted dunder-mains vs entry point crafted ones.  I
> know that in my own applications, I tend to use the latter (although
> pkg_resources performance issues bum me out).  But then you have
> applications like pex that use fairly complex hand crafted dunder-mains in
> their zip files.  In either case I don’t want consumers of my applications
> to have to worry about DeprecationWarnings, since *they* really can’t do
> anything about them.
>

This makes it the responsibility of the pex developers to at least test for
deprecation errors in their __main__. I don't know what pex is, but
presumably they have some QA and they can test their zips or at least their
__main__ with specific Python versions before distributing them. I am
confident that it's not going to be a problem for pex developers or users.


> All that to say I really don’t know what the right thing to do here is.
> All of our fiddling with the reporting of DeprecationWarnings, not to
> mention PendingDeprecationWarnings and FutureWarnings feels like
> experimental shots in the dark, and I suspect we won’t really know if PEP
> 565 will be helpful, harmful, or neutral until it’s out in the wild for a
> while.  I suspect either that what we’re trying to accomplish really can’t
> be done, or that we really don’t have a good understanding of the problem
> and we’re just chipping away at the edges.
>
> I know that’s unhelpful in deciding whether to accept the PEP or not.  In
> the absence of any clear consensus, I’m happy to trust Guido’s instincts or
> keep the status quo.
>

I also expect that this PEP will only have a small effect. It is a
compromise, but I'm okay with that. There seems to be no way to find out
what effect changes in this area will really have, because small-scale
experiments where some development team starts paying attention to
deprecations don't translate into what it will mean for the large majority
who aren't particularly interested in them (but who may still be affected
when the deprecations finally take effect and some feature disappears).

The main category of users who are going to be affected are "casual" users
-- it's they who have the most code in __main__ files and at the same time
have the least idea of where Python is header. Yes, they will occasionally
be annoyed. But they will also occasionally be glad that we point out a
deprecation to them. And, unlike the situation before Python 2.7, they
won't be annoyed by warnings in code they can't update -- they'll get
warnings only about their own scripts.

All in all, I think that for professional developers and users of
professionally developed Python packages, at worst not much will change and
at best there will be some small benefits; while for casual developers and
users there will be some benefits and those will outweigh the downsides.

In 5 years or so we can reevaluate.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Barry Warsaw]

On Dec 5, 2017, at 13:24, Guido van Rossum  wrote:

> But the whole point of the PEP is that it only warns about deprecations in 
> code over which the user has control -- likely __main__ is their own code, 
> and they *can* handle it.

I’m not so sure how true that is.  I have no sense of the relative popularity 
of hand crafted dunder-mains vs entry point crafted ones.  I know that in my 
own applications, I tend to use the latter (although pkg_resources performance 
issues bum me out).  But then you have applications like pex that use fairly 
complex hand crafted dunder-mains in their zip files.  In either case I don’t 
want consumers of my applications to have to worry about DeprecationWarnings, 
since *they* really can’t do anything about them.

All that to say I really don’t know what the right thing to do here is.  All of 
our fiddling with the reporting of DeprecationWarnings, not to mention 
PendingDeprecationWarnings and FutureWarnings feels like experimental shots in 
the dark, and I suspect we won’t really know if PEP 565 will be helpful, 
harmful, or neutral until it’s out in the wild for a while.  I suspect either 
that what we’re trying to accomplish really can’t be done, or that we really 
don’t have a good understanding of the problem and we’re just chipping away at 
the edges.

I know that’s unhelpful in deciding whether to accept the PEP or not.  In the 
absence of any clear consensus, I’m happy to trust Guido’s instincts or keep 
the status quo.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Antoine Pitrou]

On Tue, 5 Dec 2017 19:46:39 +0100
Stephane Wirtel via Python-Dev  wrote:
> But seriously, I was surprised by the number of Pull Requests and by the
> number of contributors from Feb 2017 to Oct 2017.
> 
> Here is my graph for October and November 2017.

Is it possible to compare those numbers with the number of commits /
month before the Github migration?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Stephane Wirtel via Python-Dev]

Hi Mariatta,

Thank you, I was really happy to see you at my talk, usually this kind
of talk is boring ;-) just kidding, but usually I prefer a technical talk.



Le 05/12/17 à 16:25, Mariatta Wijaya a écrit :
> I saw the talk in person :) Congrats Stéphane!
> 
> You can get the reviews from a specific PR using the API:
> https://developer.github.com/v3/pulls/reviews/#list-reviews-on-a-pull-request
> 
> For example, for reviews made to CPython PR number 1:
> 
> https://api.github.com/repos/python/cpython/pulls/1/reviews
> 
> * Time to merge a PR: 3 days in average, good!
> 
> 
> Regarding the average time to merge PR, I'm interested to know the
> average time to merge for PRs not made by Python Core Devs.
+1 I could add this point in my scripts.

Have a nice day and thank you for your feedback.

Stéphane
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Stephane Wirtel via Python-Dev]

Hi,

Thank you for this post to python-dev.

About my talk, it was a real pleasure to give it at PyCon Canada, and I
hope I could propose it to PyCon US for a larger public.

But the goal behind this talk was to show that we have a good community,
firstly by the external contributors and by the core-dev.

For the statistics, I used the REST API v3 of GitHub and matplotlib.

For the future, I would like to update them via a service running on my
own server and maybe submit it to the Python Software Foundation,
because I think it's a good indicator for the future contributors of the
project.

But seriously, I was surprised by the number of Pull Requests and by the
number of contributors from Feb 2017 to Oct 2017.

Here is my graph for October and November 2017.

I will share my scripts on Github and if you want to help me with some
good ideas, you are welcome.

Stéphane

Le 05/12/17 à 15:25, Victor Stinner a écrit :
> Hi,
> 
> Stéphane Wirtel gave a talk last month at Pycon CA about CPython pull
> requests. His slides:
> 
>https://speakerdeck.com/matrixise/cpython-loves-your-pull-requests
> 
> He produced interesting statistics that we didn't have before on pull
> requests (PR), from February 2017 to October 2017:
> 
> * total number of merged PR: 4204
> * number of contributors: 586 !!! (96%)
> * number of core developers: 27 (4%)
> * Time to merge a PR: 3 days in average, good!
> * etc.
> 
> It would be nice to get these statistics updated regularly on a
> service running somewhere.
> 
> By the way, I'm also looking for statistics on reviews on GitHub. Does
> someone know how to do that?
> 
> Victor
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Guido van Rossum]

On Tue, Dec 5, 2017 at 9:59 AM, Victor Stinner 
wrote:

> 2017-12-05 16:50 GMT+01:00 Guido van Rossum :
> > Honestly, I didn't completely follow what Victor thinks of the PEP -- his
> > post seemed mostly about promoting his own -X dev flag.
>
> -X dev is similar (but different) than -W default: show warnings which
> are hidden by default otherwise. -W default works on Python 2.7 and
> 3.6.
>
> > I have nothing
> > against that flag but I don't see how its existence is relevant to the
> PEP,
> > which is about giving users who don't even know they are Python
> developers a
> > hint when they are using deprecated features (for which there always
> must be
> > a shiny new replacement!).
>
> I disagree that *users* of an application is supposed to "handle"
> deprecation warnings: report them to the developer, or even try to fix
> them. IHMO these warnings (hidden by default) were introduced for
> developers of the application.
>

But the whole point of the PEP is that it only warns about deprecations in
code over which the user has control -- likely __main__ is their own code,
and they *can* handle it.


> My point is that I prefer to keep the status quo: continue to hide
> deprecation warnings, but promote existing solutions like -W default
> to display these warnings, teach to developers how to see and fix
> these warnings.
>

If they import a 3rd party module which does something deprecated, the user
won't see any deprecation warnings.


> Even for developers, I'm not sure that only showing warnings in
> __main__ is useful, since more and more application use a __main__
> module which is a thin entry point : import + function call (ex: "from
> app import main; main()").
>

And that's intentional -- such developers are supposed to actively test
their code, and deprecations will be shown to them by the unittest
framework. Having a minimal __main__ implies that their users won't see any
deprecation warnings.


> > Therefore I am planning to accept it by the end of this week unless more
> objections are voiced.
>
> It's ok if we disagree. I just wanted to share my opinion on this issue ;-)
>

I just worry that your opinion might be based on a misunderstanding of the
proposal. If we agree on what the PEP actually proposed to change, and how
that will affect different categories of users and developers, I am okay
with disagreement.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Victor Stinner]

2017-12-05 16:50 GMT+01:00 Guido van Rossum :
> Honestly, I didn't completely follow what Victor thinks of the PEP -- his
> post seemed mostly about promoting his own -X dev flag.

-X dev is similar (but different) than -W default: show warnings which
are hidden by default otherwise. -W default works on Python 2.7 and
3.6.

> I have nothing
> against that flag but I don't see how its existence is relevant to the PEP,
> which is about giving users who don't even know they are Python developers a
> hint when they are using deprecated features (for which there always must be
> a shiny new replacement!).

I disagree that *users* of an application is supposed to "handle"
deprecation warnings: report them to the developer, or even try to fix
them. IHMO these warnings (hidden by default) were introduced for
developers of the application.

My point is that I prefer to keep the status quo: continue to hide
deprecation warnings, but promote existing solutions like -W default
to display these warnings, teach to developers how to see and fix
these warnings.

Even for developers, I'm not sure that only showing warnings in
__main__ is useful, since more and more application use a __main__
module which is a thin entry point : import + function call (ex: "from
app import main; main()").

> Therefore I am planning to accept it by the end of this week unless more 
> objections are voiced.

It's ok if we disagree. I just wanted to share my opinion on this issue ;-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Brett Cannon]

On Tue, 5 Dec 2017 at 08:31 Senthil Kumaran  wrote:

> On Tue, Dec 5, 2017 at 6:25 AM, Victor Stinner 
> wrote:
>
>> Stéphane Wirtel gave a talk last month at Pycon CA about CPython pull
>> requests. His slides:
>>
>>https://speakerdeck.com/matrixise/cpython-loves-your-pull-requests
>>
>> He produced interesting statistics that we didn't have before on pull
>> requests (PR), from February 2017 to October 2017:
>>
>> * total number of merged PR: 4204
>> * number of contributors: 586 !!! (96%)
>> * number of core developers: 27 (4%)
>> * Time to merge a PR: 3 days in average, good!
>> * etc.
>>
>
> Plus, the slides were entertaining.
> Congrats and thanks for those stats,  Stéphane.
>

Like Mariatta I was also in the audience and Stephane was entertaining
himself as he really got into it (although he also gave me a bottle of
Belgian white wine in the talk so I may be biased :) .
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Senthil Kumaran]

On Tue, Dec 5, 2017 at 6:25 AM, Victor Stinner 
wrote:

> Stéphane Wirtel gave a talk last month at Pycon CA about CPython pull
> requests. His slides:
>
>https://speakerdeck.com/matrixise/cpython-loves-your-pull-requests
>
> He produced interesting statistics that we didn't have before on pull
> requests (PR), from February 2017 to October 2017:
>
> * total number of merged PR: 4204
> * number of contributors: 586 !!! (96%)
> * number of core developers: 27 (4%)
> * Time to merge a PR: 3 days in average, good!
> * etc.
>

Plus, the slides were entertaining.
Congrats and thanks for those stats,  Stéphane.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 540: Add a new UTF-8 mode

[Victor Stinner]

Hi,

Since it's the PEP Acceptance Week, I try my luck! Here is my very
long PEP to propose a tiny change. The PEP is very long to explain the
rationale and limitations.

Inaccurate tl; dr with the UTF-8 mode, Unicode "just works" as expected.

Reminder: INADA Naoki was nominated as the BDFL-Delegate.

https://www.python.org/dev/peps/pep-0540/

Full-text below.

Victor


PEP: 540
Title: Add a new UTF-8 mode
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner ,
Nick Coghlan 
BDFL-Delegate: INADA Naoki
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 5-January-2016
Python-Version: 3.7


Abstract


Add a new UTF-8 mode, enabled by default in the POSIX locale, to ignore
the locale and force the usage of the UTF-8 encoding for external
operating system interfaces, including the standard IO streams.

Essentially, the UTF-8 mode behaves as Python 2 and other C based
applications on \*nix systems: it aims to process text as best it can,
but it errs on the side of producing or propagating mojibake to
subsequent components in a processing pipeline rather than requiring
strictly valid encodings at every step in the process.

The UTF-8 mode can be configured as strict to reduce the risk of
producing or propagating mojibake.

A new ``-X utf8`` command line option and ``PYTHONUTF8`` environment
variable are added to explicitly control the UTF-8 mode (including
turning it off entirely, even in the POSIX locale).


Rationale
=

"It's not a bug, you must fix your locale" is not an acceptable answer
--

Since Python 3.0 was released in 2008, the usual answer to users getting
Unicode errors is to ask developers to fix their code to handle Unicode
properly. Most applications and Python modules were fixed, but users
kept reporting Unicode errors regularly: see the long list of issues in
the `Links`_ section below.

In fact, a second class of bugs comes from a locale which is not properly
configured. The usual answer to such a bug report is: "it is not a bug,
you must fix your locale".

Technically, the answer is correct, but from a practical point of view,
the answer is not acceptable. In many cases, "fixing the issue" is a
hard task. Moreover, sometimes, the usage of the POSIX locale is
deliberate.

A good example of a concrete issue are build systems which create a
fresh environment for each build using a chroot, a container, a virtual
machine or something else to get reproducible builds. Such a setup
usually uses the POSIX locale.  To get 100% reproducible builds, the
POSIX locale is a good choice: see the `Locales section of
reproducible-builds.org
`_.

PEP 538 lists additional problems related to the use of Linux containers to
run network services and command line applications.

UNIX users don't expect Unicode errors, since the common command lines
tools like ``cat``, ``grep`` or ``sed`` never fail with Unicode errors -
they produce mostly-readable text instead.

These users similarly expect that tools written in Python 3 (including
those updated from Python 2), continue to tolerate locale
misconfigurations and avoid bothering them with text encoding details.
From their point of the view, the bug is not their locale but is
obviously Python 3 ("Everything else works, including Python 2, so
what's wrong with Python 3?").

Since Python 2 handles data as bytes, similar to system utilities
written in C and C++, it's rarer in Python 2 compared to Python 3 to get
explicit Unicode errors. It also contributes significantly to why many
affected users perceive Python 3 as the root cause of their Unicode
errors.

At the same time, the stricter text handling model was deliberately
introduced into Python 3 to reduce the frequency of data corruption bugs
arising in production services due to mismatched assumptions regarding
text encodings.  It's one thing to emit mojibake to a user's terminal
while listing a directory, but something else entirely to store that in
a system manifest in a database, or to send it to a remote client
attempting to retrieve files from the system.

Since different group of users have different expectations, there is no
silver bullet which solves all issues at once. Last but not least,
backward compatibility should be preserved whenever possible.

Locale and operating system data


.. _operating system data:

Python uses an encoding called the "filesystem encoding" to decide how
to encode and decode data from/to the operating system:

* file content
* command line arguments: ``sys.argv``
* standard streams: ``sys.stdin``, ``sys.stdout``, ``sys.stderr``
* environment variables: ``os.environ``
* filenames: ``os.listdir(str)`` for example
* pipes: ``subprocess.Popen`` using ``subprocess.PIPE`` for example
* error messages: ``os.strerror(code)`` for example
* user and 

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Guido van Rossum]

If you ask me this PEP is not going to make everyone happy, but I think it
is an improvement, and it seems many people are in agreement or at least
don't object to it (and obviously Nick thinks it's going to be a big
improvement).

Therefore I am planning to accept it by the end of this week unless more
objections are voiced.

Honestly, I didn't completely follow what Victor thinks of the PEP -- his
post seemed mostly about promoting his own -X dev flag. I have nothing
against that flag but I don't see how its existence is relevant to the PEP,
which is about giving users who don't even know they are Python developers
a hint when they are using deprecated features (for which there always must
be a shiny new replacement!).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Mariatta Wijaya]

I saw the talk in person :) Congrats Stéphane!

You can get the reviews from a specific PR using the API:
https://developer.github.com/v3/pulls/reviews/#list-reviews-on-a-pull-request

For example, for reviews made to CPython PR number 1:

https://api.github.com/repos/python/cpython/pulls/1/reviews

* Time to merge a PR: 3 days in average, good!


Regarding the average time to merge PR, I'm interested to know the average
time to merge for PRs not made by Python Core Devs.


Mariatta Wijaya
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] "CPython loves your Pull Requests" talk by Stéphane Wirtel

[Victor Stinner]

Hi,

Stéphane Wirtel gave a talk last month at Pycon CA about CPython pull
requests. His slides:

   https://speakerdeck.com/matrixise/cpython-loves-your-pull-requests

He produced interesting statistics that we didn't have before on pull
requests (PR), from February 2017 to October 2017:

* total number of merged PR: 4204
* number of contributors: 586 !!! (96%)
* number of core developers: 27 (4%)
* Time to merge a PR: 3 days in average, good!
* etc.

It would be nice to get these statistics updated regularly on a
service running somewhere.

By the way, I'm also looking for statistics on reviews on GitHub. Does
someone know how to do that?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com