Re: [Python-Dev] Handle errors in cleanup code

[Nathaniel Smith]

On Mon, Jun 12, 2017 at 1:07 AM, Nick Coghlan  wrote:
> Aye, agreed. The key challenge we have is that we're trying to
> represent the exception state as a linked list, when what we really
> have once we start taking cleanup errors into account is an exception
> *tree*.
[...]
> P.S. trio's MultiError is also likely worth looking into in this context

Huh, yeah, this is some interesting convergent evolution.
trio.MultiError is exactly a way of representing a tree of exceptions,
though it's designed to do that for "live" exceptions rather than just
context chaining.

Briefly... the motivation here is that if you have multiple concurrent
call-stacks (threads/tasks/whatever-your-abstraction-is-called)
running at the same time, then it means you can literally have
multiple uncaught exceptions propagating out at the same time, so you
need some strategy for dealing with that. One popular option is to
avoid the problem by throwing away exceptions that propagate "too far"
(e.g., in the stdlib threading module, if an exception hits the top of
the call stack of a non-main thread, then it gets printed to stderr
and then the program continues normally). Trio takes a different
approach: its tasks are arranged in a tree, and if a child task exits
with an exception then that exception gets propagated into the parent
task. This allows us avoid throwing away exceptions, but it means that
we need some way to represent the situation when a parent task has
*multiple* live exceptions propagate into it at the same time. That's
what trio.MultiError is for.

Structurally, MultiError is just an Exception that holds a list of
child exceptions, like

   MultiError([TypeError(), OSError(), KeyboardInterrupt()])

so that they can propagate together. One design decision though is
that if you put a MultiError inside a MultiError, it *isn't*
collapsed, so it's also legal to have something like

MultiError([
TypeError(),
MultiError([OSError(), KeyboardInterrupt()]),
])

Semantically, these two MultiErrors are mostly the same; they both
represent a situation where there are 3 unhandled exceptions
propagating together. The reason for keeping the tree structure is
that if the inner MultiError propagated for a while on its own before
meeting the TypeError, then it accumulated some traceback and we need
somewhere to store that information. (This generally happens when the
task tree has multiple levels of nesting.) The other option would be
to make two copies of this part of the traceback and attach one copy
onto each of the two exceptions inside it, (a) but that's potentially
expensive, and (b) if we eventually print the traceback then it's much
more readable if we can say "here's where OSError was raised, and
where KeyboardInterrupt was raised, and here's where they traveled
together" and only print the common frames once.

There's some examples of how this works on pages 38-49 of my language
summit slides here:
https://vorpus.org/~njs/misc/trio-language-summit-2017.pdf
And here's the source for the toy example programs that I used in the
slides, in case anyone wants to play with them:
https://gist.github.com/njsmith/634b596e5765d5ed8b819a4f8e56d306

Then the other piece of the MultiError design is catching them. This
is done with a context manager called MultiError.catch, which "maps"
an exception handler (represented as a callable) over all the
exceptions that propagate through it, and then simplifies the
MultiError tree to collapse empty and singleton nodes. Since the
exceptions inside a MultiError represent independent, potentially
unrelated errors, you definitely don't want to accidentally throw away
that KeyboardInterrupt just because your code knows how to handle the
OSError. Or if you have something like MultiError([OSError(),
TypeError()]) then trio has no idea which of those exceptions might be
the error you know how to catch and handle which might be the error
that indicates some terrible bug that should abort the program, so
neither is allowed to mask the other - you have to decide how to
handle them independently.

If anyone wants to dig into it the code is here:
https://github.com/python-trio/trio/blob/master/trio/_core/_multierror.py

Anyway, compared to the __cleanup_errors__ idea:

- Both involve a collection object that holds exceptions, but in the
MultiError version the collection subclasses BaseException. One
consequence is that you can put the exception collection object
directly into __context__ or __cause__ instead of using a new
attribute.

- MultiError allows for a tree structure *within* a single collection
of parallel exceptions. (And then of course on top of that each
individual exception within the collection can have its own chain
attached.) Since the motivation for this is wanting to preserve
traceback structure accumulated while the collection was propagating,
and __cleanup_errors__ is only intended for "dead" expections that
don't propagate, this is solving an issue 

Re: [Python-Dev] IMPORTANT: Python 3.6.2 Maintenance Release Release Candidate in 3+ days (Monday 2017-06-12 12:00 UTC)

[Ned Deily]

An update on 3.6.2rc1: we have been working through a few critical and release 
blocker issues that needed to be fixed for 3.6.2.  Thanks to everyone who has 
helped with them!  At the moment, we have one security-related release blocker 
(http://bugs.python.org/issue29591) which I think we need to get addressed in 
3.6.2.  So, I'm delaying 3.6.2rc1 until we can get that resolved.  That means 
that, for the moment, changes going into the 3.6 branch will likely make in 
into 3.6.2.  I'll let you know when we are ready to tag.

Thanks!
--Ned


On Jun 8, 2017, at 23:34, Ned Deily  wrote:
> We are approaching the end of the second calendar quarter of 2017 and, 
> according to PEP 494, it's time to start producing the second maintenance 
> release for the 3.6 series.  The schedule calls for the release candidate to 
> be produced on Monday 2017-06-12 UTC.  As was the case with previous 3.6.x 
> releases, the plan is for the release candidate to be the same as the final 
> release, that is, no additional changes go in after the release candidate 
> except for any showstopper critical problems that might be discovered with 
> rc1.  So please plan to get any security fixes, bug fixes, and documentation 
> changes you think should be in 3.6.2 merged in ASAP.  The 3.6.2 final is 
> planned for two weeks following rc1, that is, on 2017-06-26.  The next 3.6 
> maintenance release (3.6.3) is planned to follow about 3 months later, so 
> most likely in 2017-09.
> 
> A reminder that the 3.6 branch will remain open for checkins throughout the 
> 3.6.2rc and final cycle.  Once 3.6.2rc1 is tagged, new commits to the 3.6 
> branch will release with 3.6.3.  As always, if you find any problem in 
> 3.6.2rc1 that you may believe should be corrected in 3.6.2, please ensure the 
> problem is documented in a new or existing open issue on bugs.python.org, 
> ensure that the Priority field of the issue is set to "release blocker", and 
> that "Python 3.6" is included in the selected Versions.  Comments or tags on 
> github Pull Requests are NOT sufficient!
> 
> Thanks again for all of your efforts in bringing 3.6 into the world and for 
> helping now to make it even better!
> 
> https://www.python.org/dev/peps/pep-0494/
> http://cpython-devguide.readthedocs.io

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Nathaniel Smith]

On Mon, Jun 12, 2017 at 6:29 AM, Stefan Ring  wrote:
>
> > Yury in the comment for PR 2108 [1] suggested more complicated code:
> >
> > do_something()
> > try:
> > do_something_other()
> > except BaseException as ex:
> > try:
> > undo_something()
> > finally:
> > raise ex
>
> And this is still bad, because it loses the back trace. The way we do it is:
>
> do_something()
> try:
> do_something_other()
> except BaseException as ex:
> tb = sys.exc_info()[2]
> try:
> undo_something()
> finally:
> raise ex, None, tb

Are you testing on python 2? On Python 3 just plain 'raise ex' seems
to give a sensible traceback for me...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Gregory P. Smith]

On Mon, Jun 12, 2017 at 1:48 PM Nathaniel Smith  wrote:

> On Jun 12, 2017 10:50, "Gregory P. Smith"  wrote:
>
>
> The problem, as with all warnings, is that it isn't the user who has
> control over the problem who sees the warning. It is the end use of an
> application on a system that sees it.
>
>
> I don't think I understand the distinction you're making here. This isn't
> like DeprecationWarnings where python is trying to tell the application
> developer that they need to fix something, and the application user gets
> caught in the crossfire. In this case there's nothing the application
> developer can do – the problem is that the end user has their system
> misconfigured, and probably should fix their .bashrc or crontab something
> before running the application. Or maybe they shouldn't and python should
> just do its thing and it's fine. But if anyone's going to do something it
> would have to be the end user, right?
>
> (I don't have an opinion on the warning itself; I'd be mildly interested
> to discover which of my systems are broken in this fashion but it doesn't
> affect me much either way. It does occur to me though that it will probably
> make some sysadmins *extremely* cranky: the ones that have crontabs with
> broken locales and which are set to email if their script generates any
> output – I think these are both still cron defaults – and who after
> upgrading to py37 will suddenly find copies of this warning pouring into
> their inboxes at a rate of hundreds per hour.)
>
> -n
>

I guess so. It is a system environment configuration problem. But I'd still
imagine that end users never touch their LANG or LC_* environment variables
themselves.  (the only time I touch anything is to explicitly set LANG=C on
occasion when I want text process shell tools to run a lot faster; but I
hardly count as I live in ascii-land, aka the US)

I'm inclined to suggest it should be silent, but I'm curious if we can get
some linux distros testing with it enabled during the beta period to see
how much impact it actually has on users in the real world. We could be
talking about a minor non-issue.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Nathaniel Smith]

On Jun 12, 2017 10:50, "Gregory P. Smith"  wrote:


The problem, as with all warnings, is that it isn't the user who has
control over the problem who sees the warning. It is the end use of an
application on a system that sees it.


I don't think I understand the distinction you're making here. This isn't
like DeprecationWarnings where python is trying to tell the application
developer that they need to fix something, and the application user gets
caught in the crossfire. In this case there's nothing the application
developer can do – the problem is that the end user has their system
misconfigured, and probably should fix their .bashrc or crontab something
before running the application. Or maybe they shouldn't and python should
just do its thing and it's fine. But if anyone's going to do something it
would have to be the end user, right?

(I don't have an opinion on the warning itself; I'd be mildly interested to
discover which of my systems are broken in this fashion but it doesn't
affect me much either way. It does occur to me though that it will probably
make some sysadmins *extremely* cranky: the ones that have crontabs with
broken locales and which are set to email if their script generates any
output – I think these are both still cron defaults – and who after
upgrading to py37 will suddenly find copies of this warning pouring into
their inboxes at a rate of hundreds per hour.)

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Glenn Linderman]

On 6/12/2017 1:11 PM, Terry Reedy wrote:
I do not have any locale-related env vars.  You should check whether 
the warning is off on all Win10 systems, as well as Win7 and Win8.  
Many Windows users know nothing about ENV VARS, and even if they do, 
they may not know how (I don't know the details) or MAY NOT BE ABLE to 
set one permanently. 


where (latest Win10): in the control panel, System and Security, System, 
Advanced system settings (left edge menu item), Advanced Tab, 
Environment Variables (button).


Two sets of variables. User should always be able to set User variables 
that are in effect for all new processes launched after creation of the 
variable. System variables may be locked down if not administrator.


Both types are permanent, stored in the registry.

From a command line, batch file, or program launcher, temporary, local 
environment variables can be set that only apply to other programs in 
that session and that session's child processes.


So there is no fear of being unable to set an environment variable on 
Windows. Naturally, some users may not know what they are or how to set 
them, but it is well-documented in various places... just google "how to 
set windows environment variables".
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Paul Moore]

On 12 June 2017 at 21:11, Terry Reedy  wrote:
> I do not see this with my fresh 3.7 repository debug build on my Win 10
> machine.  If I did, I would be -1000.  This warning should be treated
> as a deprecation warning, off by default.

My understanding is that this is a Unix-only change. If it affects
Windows, I'll add my voice to the complaints that we don't want a
warning by default. (But if it's Unix-only, I'm happy to leave the
comments to the people who are affected).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Terry Reedy]

On 6/12/2017 5:35 AM, INADA Naoki wrote:

On Mon, Jun 12, 2017 at 5:56 PM, Victor Stinner
 wrote:

Hi,

Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
forward to UTF-8 everywhere ;-)

I would prefer to not be annoyed by warning messages about encodings
at startup if possible:

"Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
behavior)."


I do not see this with my fresh 3.7 repository debug build on my Win 10 
machine.  If I did, I would be -1000.  This warning should be treated

as a deprecation warning, off by default.


This is intentional behavior, to motivate people to use UTF-8 locale without
coercion.


I thought the idea of the PEP was to do the right thing without making 
people do anything.



Note: I don't consider that 2>/dev/null is a good practice to ignore a
single warning, since it will ignore *all* message written into
stderr, including useful warnings like ResourceWarning or "Exception
ignored ...".


I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable,
as warning says.


I do not have any locale-related env vars.  You should check whether the 
warning is off on all Win10 systems, as well as Win7 and Win8.  Many 
Windows users know nothing about ENV VARS, and even if they do, they may 
not know how (I don't know the details) or MAY NOT BE ABLE to set one 
permanently.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[INADA Naoki]

>> -1 for disable coercion by default:  It's too unhelpful for beginners.
>
> Are you proposing to reject the PEP that you approved? Now I'm confused.
>

No, I just wanted to clarify your propose.

You said "I'm able to render my hello world with the wrong locale :-)".
So I want to clarify you didn't mean "Python 3.6 can print ASCII in C locale,
so stop coercion and warning by default".

Now I think I understand your point.


>
> Why would user need to be annoyed by a warning while Python just do
> the "right thing" for them?

Hmm, I didn't have enough confident about locale coercion is the "right thing"
always.

But since current PEP 538 changes only LC_CTYPE, there are very small
chance for unwanted side-effect.  I agree it's small enough to do the coercion
silently.

There are some unhappy feedback about warning already.
Unless I see some unhappy feedback about coercion, I'm +1 to remove the
warning at some point before 3.7.0.
And I change my -0.5 to -0 about removing it right now.

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Brett Cannon]

On Mon, 12 Jun 2017 at 01:08 Nick Coghlan  wrote:

> On 12 June 2017 at 17:31, Martin (gzlist) via Python-Dev
>  wrote:
> > On 12/06/2017, Serhiy Storchaka  wrote:
> >>
> >> But if an error is raised when execute undo_something(), it replaces the
> >> original exception which become chaining as the __context__ attribute.
> >> The problem is that this can change the type of the exception. If
> >> do_something_other() raises SystemExit and undo_something() raises
> >> KeyError, the final exception has type KeyError.
> >
> > For Python 2.7, I've used the following idiom, which always masks
> > errors from the undo:
> >
> >   do_something()
> >   try:
> >   do_something_other()
> >   except:
> >   try:
> >   undo_something()
> >   finally:
> >   raise
> >
> > Unfortunately, that breaks down on Python 3, as the new exception
> > propogates with the original as context..
>
> Relevant tracker issue for that problem:
> https://bugs.python.org/issue18861
>
> >> Does it mean that we should rewrite every chunk of code similar to the
> >> above? And if such cumbersome code is necessary and become common, maybe
> >> it needs syntax support in the language? Or changing the semantic of
> >> exceptions raised in error handlers and finally blocks?
> >
> > What I want is a way to chain exceptions in the other direction,
> > raising the original error, but attaching a related one. Unfortunately
> > neither __cause__ or __context__ really help.
>
> Aye, agreed. The key challenge we have is that we're trying to
> represent the exception state as a linked list, when what we really
> have once we start taking cleanup errors into account is an exception
> *tree*.
>
> The thing that finally clicked for me in this thread is that if we add
> contextlib.ResourceSet, and have it add a new attribute to the
> original exception (e.g. "__cleanup_errors__") with a list of
> exceptions raised while attempt to cleanup after the earlier
> exception, then that lets us actually start modelling that tree at
> runtime.
>

You might want to go back and read Jake's LWN article for this year's
language summit as Nathaniel Smith said he wanted some sort of
multi-exception type which would go along with this idea.

-Brett


>
> Once we understand the *behaviour* we want, *then* we can consider
> whether we might want to add a context manager to have any raised
> exceptions be attached to the currently active exception as cleanup
> errors rather than as new exceptions in their own right.
>
> For example:
>
> do_something()
> try:
> do_something_other()
> except BaseException as exc:
> with contextlib.cleanup(exc) as reraise:
> # Exceptions raised in here would be added to
> # exc.__cleanup_errors__, rather than being
> # propagated normally
> undo_something()
> reraise()
>
> The need for the "as reraise:"/"reraise()" trick arises from the need
> to special case the situation where the original exception inherits
> from Exception, but one of the raised exceptions *doesn't* - we want
> SystemExit/KeyboardInterrupt/etc to take precedence in that case, and
> a bare raise statement won't handle that for us (it *could* in a
> hypothetical future version of Python that's natively
> `__cleanup_errors__` aware, but that's not going to be useful for
> existing versions).
>
> Since I don't see anything in the discussion so far that *requires*
> changes to the standard library (aside from "we may want to use this
> ourselves"), the right place to thrash out the design details is
> probably contextlib2: https://github.com/jazzband/contextlib2
>
> That's where contextlib.ExitStack was born, and I prefer using it to
> iterate on context management design concepts, since we can push
> updates out faster, and if we make bad choices anywhere along the way,
> they can just sit around in contextlib2, rather than polluting the
> standard library indefinitely.
>
> Cheers,
> Nick.
>
> P.S. trio's MultiError is also likely worth looking into in this context
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Gregory P. Smith]

On Mon, Jun 12, 2017 at 5:25 AM Nick Coghlan  wrote:

> On 12 June 2017 at 18:56, Victor Stinner  wrote:
> > Hi,
> >
> > Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
> > forward to UTF-8 everywhere ;-)
> >
> > I would prefer to not be annoyed by warning messages about encodings
> > at startup if possible:
> >
> > "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
> > locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
> > behavior)."
>
> Note that there's an open issue for this linked from the What's New entry:
>
> *
> https://docs.python.org/dev/whatsnew/3.7.html#pep-538-legacy-c-locale-coercion
> * https://bugs.python.org/issue30565
>
> I suspect the eventual outcome is going to be dropping that particular
> warning (since it's been problematic for Fedora's 3.6 backport as
> well, and the problems are due to the warning itself, *not* the locale
> coercion), but I'd prefer to keep the notification at least for a
> while (potentially even until alpha 1).
>
> OTOH, I'm also open to being persuaded otherwise if enough folks are
> running into problems with it just while working on CPython (I'd still
> like to turn it back on for alpha 1 even if we turn off in the
> meantime, though).
>
> Cheers,
> Nick.
>
> P.S. Part of my rationale for doing it this way is that I'm certain
> that after 3.7's release next year we're going to get at least a few
> users genuinely upset at our decision to move the ASCII-based C locale
> explicitly into the "legacy partially-supported environment" category,
> and even more upset that we're "silently ignoring their explicit
> configuration settings" by implicitly coercing it to something else.
>
> Those kinds of concerns are much easier to address effectively if we
> can say "We tried it with an explicit warning, and it was too annoying
> to be usable; see  if you want more details" than if we're
> in the situation of having to say "We assumed an explicit warning
> would be too annoying, so we never even tried it".
>

The problem, as with all warnings, is that it isn't the user who has
control over the problem who sees the warning. It is the end use of an
application on a system that sees it.

So they way I see it, we have no choice but to disable this warning by
default before we exit 3.7 beta.

-gps - "infamous" author of the bad idea of a DeprecationWarning in the old
md5 and sha modules...


>
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On "PEP 546 — Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7"

[Victor Stinner]

FYI I started to work on the implementation: I rebased Alex Gaynor's
patch written in 2014 and converted it to a pull request.
http://bugs.python.org/issue22559

Victor

2017-06-10 1:56 GMT+02:00 Benjamin Peterson :
> The reason we're having this conversation at all is probably a matter of
> timing. If MemoryBIO was in Python 3 when PEP 466 was accepted, it surely
> would have come along for the ride to 2.7. I believe PEP 466 is generally
> considered to have produced positive results. PEP 546, carrying no breaking
> changes, is less risky than PEP 466.
>
> The reluctance to bend 2.7 rules is healthy. This PEP is part of the price
> we pay, though, for making a backwards-incompatible release. The security
> landscape has and will change over the 10+ python-dev-supported life span of
> 2.7. During that time, we have an obligation to keep Python 2 secure. Part
> of that is supporting modern security interfaces, which are features. This
> change is needed to make another stdlib feature, ensurepip (which is itself
> yet another 2.7.x backport) work well.
>
> Therefore, as 2.7 release manager, I'm accepting the PEP.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Ethan Furman]

On 06/12/2017 05:24 AM, Nick Coghlan wrote:


I suspect the eventual outcome is going to be dropping that particular
warning (since it's been problematic for Fedora's 3.6 backport as
well, and the problems are due to the warning itself, *not* the locale
coercion), but I'd prefer to keep the notification at least for a
while (potentially even until alpha 1).


As long as we take it out for 3.7.0 I'll be happy.  At that point it can be enabled by whichever flag we have that says 
emit all warnings instead of silently suppressing them.


This reminds me of pip with it's 20 lines of ssl and sudo warnings -- it makes it nearly unusable as I have to search 
and search to find the two lines somewhere in the middle that I'm interested in.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Victor Stinner]

2017-06-12 15:28 GMT+02:00 INADA Naoki :
>> I like using LANG=C to display a manual page in english.
>
> Me too.  But we can use LC_CTYPE=C.UTF-8 with LANG=C.

My point is that LANG=C is easy to remember and "it just works".

> And LC_CTYPE=C.UTF-8 is much easier (and ideal) than
> PYTHONLOCALECOERCIONWARNING=0.

I just propose to remove the warning, so LANG=C would "just works" as
well with Python 3.7.

>> Technically, I know that it's wrong, but it works. I don't see the point of 
>> the
>> warning. I'm able to render my hello world with the wrong locale :-)
>> If you want to use C.utf-8, fine, just do it, but don't bug me with
>> locales please :-)
>
> Do you mean which?
>
> b) don't show warning when locale is coerced.

Always remove the warning.

>> I consider that I understand well locales and encodings. But now try
>> to imagine someone who don't know anything about programming and
>> starts learning a new language, Python, and see this warning...
>
> Locale coercion is for them.  Currently (Python 3.6), they will see
> UnicodeEncodeError.
> This warning is much helpful than UnicodeEncodeError, because
> it teaches how to configure UTF-8 locale.

Sorry, I don't understand. Currently, Python 3.7 "just works": it uses
UTF-8 when the locale is C and so we don't get any UnicodeError
anymore.

Why would user need to be annoyed by a warning while Python just do
the "right thing" for them?

> And I fear about overriding locale silently.

The thing is nobody understand locales :-) No need to annoy users
about locales. Just fix them silently ;-)

>  But there may be someone who prefer strict,
> ascii-only C locale.  This warning tells how to disable coercion for them.

Users who understand locales don't need such warning. They know how to
read a documentation and how to use properly locales :-)


> -1 for disable coercion by default:  It's too unhelpful for beginners.

Are you proposing to reject the PEP that you approved? Now I'm confused.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Stefan Ring]

> Yury in the comment for PR 2108 [1] suggested more complicated code:
>
> do_something()
> try:
> do_something_other()
> except BaseException as ex:
> try:
> undo_something()
> finally:
> raise ex

And this is still bad, because it loses the back trace. The way we do it is:

do_something()
try:
do_something_other()
except BaseException as ex:
tb = sys.exc_info()[2]
try:
undo_something()
finally:
raise ex, None, tb
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[INADA Naoki]

On Mon, Jun 12, 2017 at 6:46 PM, Victor Stinner
 wrote:
> 2017-06-12 11:35 GMT+02:00 INADA Naoki :
>> I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable,
>> as warning says.
>
> I like using LANG=C to display a manual page in english.

Me too.  But we can use LC_CTYPE=C.UTF-8 with LANG=C.

And LC_CTYPE=C.UTF-8 is much easier (and ideal) than
PYTHONLOCALECOERCIONWARNING=0.

> Technically, I know that it's wrong, but it works. I don't see the point of 
> the
> warning. I'm able to render my hello world with the wrong locale :-)
> If you want to use C.utf-8, fine, just do it, but don't bug me with
> locales please :-)

Do you mean which?

a) make locale coercion off by default.
b) don't show warning when locale is coerced.


> I consider that I understand well locales and encodings. But now try
> to imagine someone who don't know anything about programming and
> starts learning a new language, Python, and see this warning...

Locale coercion is for them.  Currently (Python 3.6), they will see
UnicodeEncodeError.
This warning is much helpful than UnicodeEncodeError, because
it teaches how to configure UTF-8 locale.

And I fear about overriding locale silently.  AFAIK, locale coercion is
very nice for most cases.  But there may be someone who prefer strict,
ascii-only C locale.  This warning tells how to disable coercion for them.

And there may be some surprising side effect of coercing location
we can't expect for now.  That's another reason why coercion shows warning.

But we use `surrogateescape` error handler for stdout in C locale already.
I may be too nervous about it.

So my vote is:

-1 for disable coercion by default:  It's too unhelpful for beginners.
-0.5 for disable warning NOW:  I don't think we know all side effects well.


Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Joao S. O. Bueno]

On 12 June 2017 at 14:24, Nick Coghlan  wrote:
> On 12 June 2017 at 18:56, Victor Stinner  wrote:
>> Hi,
>>
>> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
>> forward to UTF-8 everywhere ;-)
>>
>> I would prefer to not be annoyed by warning messages about encodings
>> at startup if possible:
>>
>> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
>> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
>> behavior)."
>
> Note that there's an open issue for this linked from the What's New entry:
>
> * 
> https://docs.python.org/dev/whatsnew/3.7.html#pep-538-legacy-c-locale-coercion
> * https://bugs.python.org/issue30565
>
> I suspect the eventual outcome is going to be dropping that particular
> warning (since it's been problematic for Fedora's 3.6 backport as
> well, and the problems are due to the warning itself, *not* the locale
> coercion), but I'd prefer to keep the notification at least for a
> while (potentially even until alpha 1).
>
> OTOH, I'm also open to being persuaded otherwise if enough folks are
> running into problems with it just while working on CPython (I'd still
> like to turn it back on for alpha 1 even if we turn off in the
> meantime, though).
>
> Cheers,
> Nick.
>
> P.S. Part of my rationale for doing it this way is that I'm certain
> that after 3.7's release next year we're going to get at least a few
> users genuinely upset at our decision to move the ASCII-based C locale
> explicitly into the "legacy partially-supported environment" category,
> and even more upset that we're "silently ignoring their explicit
> configuration settings" by implicitly coercing it to something else.


Yes - all these 15 users can be quite noisy - they are not quite users of
"whatever there is is good" like the other 800 million or so users
that will be bothered by
the warning.
(Ok -  I guess most of the 800 million users won't even be seeing a
terminal when running their Python -
but still it would be a  couple million users and let's suppose there
are 150 and not 15 genuinely worried
about that coercion)


>
> Those kinds of concerns are much easier to address effectively if we
> can say "We tried it with an explicit warning, and it was too annoying
> to be usable; see  if you want more details" than if we're
> in the situation of having to say "We assumed an explicit warning
> would be too annoying, so we never even tried it".

Perfect - and guess what? It looks like it already happened, as you
can see by these e-mail messages.
Therefore we are good to remove the warning now.

>
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Nick Coghlan]

On 12 June 2017 at 18:56, Victor Stinner  wrote:
> Hi,
>
> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
> forward to UTF-8 everywhere ;-)
>
> I would prefer to not be annoyed by warning messages about encodings
> at startup if possible:
>
> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
> behavior)."

Note that there's an open issue for this linked from the What's New entry:

* https://docs.python.org/dev/whatsnew/3.7.html#pep-538-legacy-c-locale-coercion
* https://bugs.python.org/issue30565

I suspect the eventual outcome is going to be dropping that particular
warning (since it's been problematic for Fedora's 3.6 backport as
well, and the problems are due to the warning itself, *not* the locale
coercion), but I'd prefer to keep the notification at least for a
while (potentially even until alpha 1).

OTOH, I'm also open to being persuaded otherwise if enough folks are
running into problems with it just while working on CPython (I'd still
like to turn it back on for alpha 1 even if we turn off in the
meantime, though).

Cheers,
Nick.

P.S. Part of my rationale for doing it this way is that I'm certain
that after 3.7's release next year we're going to get at least a few
users genuinely upset at our decision to move the ASCII-based C locale
explicitly into the "legacy partially-supported environment" category,
and even more upset that we're "silently ignoring their explicit
configuration settings" by implicitly coercing it to something else.

Those kinds of concerns are much easier to address effectively if we
can say "We tried it with an explicit warning, and it was too annoying
to be usable; see  if you want more details" than if we're
in the situation of having to say "We assumed an explicit warning
would be too annoying, so we never even tried it".


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale

[Nick Coghlan]

On 12 June 2017 at 17:47, Martin (gzlist)  wrote:
> Thanks for replying to my points!
>
> On 12/06/2017, Nick Coghlan  wrote:
>>
>> `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict
>> encoding checks on the standard streams, regardless of locale.
>
> Then the user of my script has to care that it's written in Python and
> set that specifically in their crontab or so on...

As Inada-san wrote, we think the right way to fix that is to make it
easier and safer for application developers to override the default
settings on the standard streams. At the moment, doing so requires
rebinding sys.stdin/out/err, which means you end up with multiple
Python level streams sharing the one underlying C stream, which can
cause problems.

The basic API for that was recently merged
(`TextIOWrapper.reconfigure()`), so it's now a matter of extending it
to also allow updating `encoding` and `errors`.

>> In addition to providing a reliable escape hatch with no other
>> potentially unwanted side effects (for when folks actually want the
>> current behaviour), the entry for the off switch in the CLI usage docs
>> also provides us with a convenient place to document the *default*
>> behaviour.
>
> The documentation aspect is an interesting consideration.
>
> Having thought about it a bit more, my preferred option is having the
> disable be if either LC_ALL or LC_CTYPE vars are exactly 'C', then
> don't override. Otherwise (including for LANG=C), force C.UTF-8. The
> CLI usage docs could have a LC_CTYPE entry that goes into details of
> why.

LC_ALL=C doesn't actually disable the locale coercion (i.e. we still
set LC_CTYPE). The coercion just doesn't have any effect, since LC_ALL
takes precedence.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Victor Stinner]

2017-06-12 11:35 GMT+02:00 INADA Naoki :
> I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable,
> as warning says.

I like using LANG=C to display a manual page in english. Technically,
I know that it's wrong, but it works. I don't see the point of the
warning. I'm able to render my hello world with the wrong locale :-)
If you want to use C.utf-8, fine, just do it, but don't bug me with
locales please :-)

I consider that I understand well locales and encodings. But now try
to imagine someone who don't know anything about programming and
starts learning a new language, Python, and see this warning...

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[INADA Naoki]

On Mon, Jun 12, 2017 at 5:56 PM, Victor Stinner
 wrote:
> Hi,
>
> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
> forward to UTF-8 everywhere ;-)
>
> I would prefer to not be annoyed by warning messages about encodings
> at startup if possible:
>
> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
> behavior)."
>

This is intentional behavior, to motivate people to use UTF-8 locale without
coercion.

> Note: I don't consider that 2>/dev/null is a good practice to ignore a
> single warning, since it will ignore *all* message written into
> stderr, including useful warnings like ResourceWarning or "Exception
> ignored ...".

I think "Good practice" is set `LC_CTYPE=C.UTF-8` environment variable,
as warning says.

# When PEP 540 is accepted, it can be used to disable the warning too.

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 warning at startup: please remove it

[Antoine Pitrou]

On Mon, 12 Jun 2017 10:56:31 +0200
Victor Stinner  wrote:
> Hi,
> 
> Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
> forward to UTF-8 everywhere ;-)
> 
> I would prefer to not be annoyed by warning messages about encodings
> at startup if possible:
> 
> "Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
> locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
> behavior)."

+1 for muting this warning.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 538 warning at startup: please remove it

[Victor Stinner]

Hi,

Nick Coghlan pushed his implementation of his PEP 538: nice! Nice step
forward to UTF-8 everywhere ;-)

I would prefer to not be annoyed by warning messages about encodings
at startup if possible:

"Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
behavior)."


Python 3.7 is the only programming language that I know that complains
about encoding at startup. Even if my code is 100% valid, PEP 8
compliant, don't emit any warning, etc. I will get the warning. It is
*not* possible to ignore the warning... Like "yeah, I know that my
locale is C, but it's not like I'm able to configure it."

haypo@selma$ export LANG=
haypo@selma$ locale
LANG=
LC_CTYPE="POSIX"
...

haypo@selma$ ./python -c 'print("Hello World!")' # Python 3.7
Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
behavior).
Hello World!

haypo@selma$ python2 -c 'print("Hello World!")' # Python 2.7
Hello World!

haypo@selma$ perl -e 'print "Hello, world!\n"' # Perl 5.24
Hello, world!

haypo@selma$ ./c   # C
Hello World!

...

Note: I don't consider that 2>/dev/null is a good practice to ignore a
single warning, since it will ignore *all* message written into
stderr, including useful warnings like ResourceWarning or "Exception
ignored ...".

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale

[INADA Naoki]

> On 12/06/2017, Nick Coghlan  wrote:
>>
>> `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict
>> encoding checks on the standard streams, regardless of locale.
>
> Then the user of my script has to care that it's written in Python and
> set that specifically in their crontab or so on...
>

That's why I think https://bugs.python.org/issue15216 should be fixed
in Python 3.7 too.

Python should have one preferable way to specify encoding and error
handler from inside of the program, not from envvar or command line argument.

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Nick Coghlan]

On 12 June 2017 at 17:31, Martin (gzlist) via Python-Dev
 wrote:
> On 12/06/2017, Serhiy Storchaka  wrote:
>>
>> But if an error is raised when execute undo_something(), it replaces the
>> original exception which become chaining as the __context__ attribute.
>> The problem is that this can change the type of the exception. If
>> do_something_other() raises SystemExit and undo_something() raises
>> KeyError, the final exception has type KeyError.
>
> For Python 2.7, I've used the following idiom, which always masks
> errors from the undo:
>
>   do_something()
>   try:
>   do_something_other()
>   except:
>   try:
>   undo_something()
>   finally:
>   raise
>
> Unfortunately, that breaks down on Python 3, as the new exception
> propogates with the original as context..

Relevant tracker issue for that problem: https://bugs.python.org/issue18861

>> Does it mean that we should rewrite every chunk of code similar to the
>> above? And if such cumbersome code is necessary and become common, maybe
>> it needs syntax support in the language? Or changing the semantic of
>> exceptions raised in error handlers and finally blocks?
>
> What I want is a way to chain exceptions in the other direction,
> raising the original error, but attaching a related one. Unfortunately
> neither __cause__ or __context__ really help.

Aye, agreed. The key challenge we have is that we're trying to
represent the exception state as a linked list, when what we really
have once we start taking cleanup errors into account is an exception
*tree*.

The thing that finally clicked for me in this thread is that if we add
contextlib.ResourceSet, and have it add a new attribute to the
original exception (e.g. "__cleanup_errors__") with a list of
exceptions raised while attempt to cleanup after the earlier
exception, then that lets us actually start modelling that tree at
runtime.

Once we understand the *behaviour* we want, *then* we can consider
whether we might want to add a context manager to have any raised
exceptions be attached to the currently active exception as cleanup
errors rather than as new exceptions in their own right.

For example:

do_something()
try:
do_something_other()
except BaseException as exc:
with contextlib.cleanup(exc) as reraise:
# Exceptions raised in here would be added to
# exc.__cleanup_errors__, rather than being
# propagated normally
undo_something()
reraise()

The need for the "as reraise:"/"reraise()" trick arises from the need
to special case the situation where the original exception inherits
from Exception, but one of the raised exceptions *doesn't* - we want
SystemExit/KeyboardInterrupt/etc to take precedence in that case, and
a bare raise statement won't handle that for us (it *could* in a
hypothetical future version of Python that's natively
`__cleanup_errors__` aware, but that's not going to be useful for
existing versions).

Since I don't see anything in the discussion so far that *requires*
changes to the standard library (aside from "we may want to use this
ourselves"), the right place to thrash out the design details is
probably contextlib2: https://github.com/jazzband/contextlib2

That's where contextlib.ExitStack was born, and I prefer using it to
iterate on context management design concepts, since we can push
updates out faster, and if we make bad choices anywhere along the way,
they can just sit around in contextlib2, rather than polluting the
standard library indefinitely.

Cheers,
Nick.

P.S. trio's MultiError is also likely worth looking into in this context

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale

[Martin (gzlist) via Python-Dev]

Thanks for replying to my points!

On 12/06/2017, Nick Coghlan  wrote:
>
> `PYTHONIOENCODING=:strict` remains the preferred way of forcing strict
> encoding checks on the standard streams, regardless of locale.

Then the user of my script has to care that it's written in Python and
set that specifically in their crontab or so on...

> In addition to providing a reliable escape hatch with no other
> potentially unwanted side effects (for when folks actually want the
> current behaviour), the entry for the off switch in the CLI usage docs
> also provides us with a convenient place to document the *default*
> behaviour.

The documentation aspect is an interesting consideration.

Having thought about it a bit more, my preferred option is having the
disable be if either LC_ALL or LC_CTYPE vars are exactly 'C', then
don't override. Otherwise (including for LANG=C), force C.UTF-8. The
CLI usage docs could have a LC_CTYPE entry that goes into details of
why.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Nick Coghlan]

On 12 June 2017 at 17:10, Serhiy Storchaka  wrote:
> Does it mean that we should rewrite every chunk of code similar to the
> above? And if such cumbersome code is necessary and become common, maybe it
> needs syntax support in the language? Or changing the semantic of exceptions
> raised in error handlers and finally blocks?

No, as in general there's no way for us to reliably tell the
difference between intentionally changing the exception type (e.g.
KeyError->AttributeError and vice-versa), unintentionally changing the
exception type due to a bug in the cleanup code (e.g.
SystemExit->KeyError), and the exception type changing due to external
events (e.g. KeyError -> KeyboardInterrupt).

So in the example given if an expected-and-allowed kind of exception
can escape from "undo_something()", it's not really cleanup code, it's
normal code, and needs to be refactored more like the way most close()
methods work: particular known-to-be-possible exceptions get caught
and reported (or silently ignored), but not propagated.

That last example also shows that the suggested replacement idiom is
incorrect, as it can suppress SystemExit, KeyboardInterrupt, etc.

That said, as far as better builtin support for dynamic resource
management goes: that would currently be contextlib.ExitStack.

I'm also completely open to adding more hooks to let users tweak how
that reports exceptions, including adding a contextlib.ResourceSet
variant, which *doesn't* semantically nest the callback handling the
way ExitStack does, and instead collects any exceptions raised into a
new __cleanup_errors__ attribute on the original exception (thus
leaving the original exception chain untouched).

That would still need some fiddling to work out what to do in the
"exception that doesn't inherit from Exception" case (probably clean
up all the registered resources anyway then raise the first such
exception caught, while still attaching all the others to
__cleanup_errors__ on the original exception), but it's likely to be
significantly more feasible than doing anything at the syntax level.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handle errors in cleanup code

[Martin (gzlist) via Python-Dev]

On 12/06/2017, Serhiy Storchaka  wrote:
>
> But if an error is raised when execute undo_something(), it replaces the
> original exception which become chaining as the __context__ attribute.
> The problem is that this can change the type of the exception. If
> do_something_other() raises SystemExit and undo_something() raises
> KeyError, the final exception has type KeyError.

For Python 2.7, I've used the following idiom, which always masks
errors from the undo:

  do_something()
  try:
  do_something_other()
  except:
  try:
  undo_something()
  finally:
  raise

Unfortunately, that breaks down on Python 3, as the new exception
propogates with the original as context..

> Does it mean that we should rewrite every chunk of code similar to the
> above? And if such cumbersome code is necessary and become common, maybe
> it needs syntax support in the language? Or changing the semantic of
> exceptions raised in error handlers and finally blocks?

What I want is a way to chain exceptions in the other direction,
raising the original error, but attaching a related one. Unfortunately
neither __cause__ or __context__ really help.

Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Handle errors in cleanup code

[Serhiy Storchaka]

There is an idiomatic Python code:

do_something()
try:
do_something_other()
except:
undo_something()
raise

If an error is raised when execute do_something_other(), then we should 
first restore the state that was before calling do_something(), and then 
reraise the exception. It is important that the bare "except" (or 
"except BaseException") is used, because undo_something() should be 
executed on any exception. And this is one of few cases where using the 
bare "except" is correct.


But if an error is raised when execute undo_something(), it replaces the 
original exception which become chaining as the __context__ attribute. 
The problem is that this can change the type of the exception. If 
do_something_other() raises SystemExit and undo_something() raises 
KeyError, the final exception has type KeyError.


Yury in the comment for PR 2108 [1] suggested more complicated code:

do_something()
try:
do_something_other()
except BaseException as ex:
try:
undo_something()
finally:
raise ex

Does it mean that we should rewrite every chunk of code similar to the 
above? And if such cumbersome code is necessary and become common, maybe 
it needs syntax support in the language? Or changing the semantic of 
exceptions raised in error handlers and finally blocks?


[1] https://github.com/python/cpython/pull/2108

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com