Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On 2019-02-22 17:20, Chris Angelico wrote: > On Sat, Feb 23, 2019 at 9:14 AM Kyle Lahnakoski > wrote: >> Can Python provide better support for the CNCR pattern? If it is >> lightweight enough, maybe people will use it, and then we can say >> something useful about the (restricted) range of exceptions coming from >> a method: > The CNCR pattern, if used repeatedly, will quickly create a long chain > of exceptions, where each exception represents one function call. > Python already has very good support for seeing the function call > history that led to the exception - it's called a traceback. You even > get the full function locals as part of the exception object... and it > requires no code whatsoever! Simply allowing exceptions to bubble up > will have practically the same benefit. I like your point that the exception chain is long, and similar to the stack: I do find that in practice. This may indicate there is an optimization in how CNCR can be handled: Maybe instantiation of CNCR exceptions can be deferred to the point where an exception handler actually does something beyond CNCR, if ever. I also agree, with a debugger at hand, we can inspect the stack trace. We can also write code that reflects on the method names in the stack trace to figure out how to handle an exception. But, I am concerned that stack trace inspection is brittle because methods can be renamed and refactored. Also, the CNCR pattern is not 1-1 with methods, there can be more than one type of CNCR exception emitted from a method. Methods could be split, so each only throws one type of exception; and then the stack trace would suffice; but that brings us back to brittle: A split method may be accidentally merged for clarity at a later time. We should also consider what happens in the case that an exception chain is not handled: It may be printed to the log: We can not reasonably print all the locals; there are many, some do not serialize, and some are sensitive. CNCR is being explicit about what locals(), if any, are important. ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Sat, Feb 23, 2019 at 9:14 AM Kyle Lahnakoski wrote:
> Let me call this pattern the Catch-It-Name-It-Chain-It-Raise-It (CNCR)
> pattern
>
> There are a few reasons for this.
>
> 1. I can add runtime values to the exception so I get a better sense of
> the program state without going to the debugger: `error("some
> description", {"url": url}, cause=e)`
> 2. I prevent exception leakage; I have no idea the diversity of
> exceptions my code can raise, so I CNCR.
> 3. Every exception is it's own unique type; I can switch on the message
> if I want (I rarely do this, but it happens, see below)
>
> Can Python provide better support for the CNCR pattern? If it is
> lightweight enough, maybe people will use it, and then we can say
> something useful about the (restricted) range of exceptions coming from
> a method:
The CNCR pattern, if used repeatedly, will quickly create a long chain
of exceptions, where each exception represents one function call.
Python already has very good support for seeing the function call
history that led to the exception - it's called a traceback. You even
get the full function locals as part of the exception object... and it
requires no code whatsoever! Simply allowing exceptions to bubble up
will have practically the same benefit.
ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On 2019-02-21 03:09, Christopher Barker wrote:
>
> But yes, there is no (easy) way to distinguish an Exception raised by
the function you called, and one raised somewhere deeper that.
>
> And I have been bitten by that more than once. It makes "Easier to ask
forgiveness than permission" kind of tricky.
> And Exception handling is messy -- the point made by the OP, I'm not
sure there's a better way to do it.
It seems to me that exception *classes* are hardly ever required. If two
methods raise the same exception type, then I contend that they are
actually throwing two different types of errors (as evidenced by the
difference in the message). Instead, I suggest every method emits its
own exception type.
By demanding support for a plethora of exception types, at least as many
as there are methods, then we may have a different way of looking at
exceptions: The first conclusion is, it is too expensive to make classes
for every exception type, rather, exception type should be defined by
the message.
Exception handling is messy. I find most of my Python methods look like:
| def my_method():
| try:
| # do something
| except Exception as e:
| error("some description", cause=e)
I am not using Python3 everywhere yet, so maybe it should read like:
| def my_method():
| try:
| # do something
| except Exception as e:
| raise Exception("some description") from e
Let me call this pattern the Catch-It-Name-It-Chain-It-Raise-It (CNCR)
pattern
There are a few reasons for this.
1. I can add runtime values to the exception so I get a better sense of
the program state without going to the debugger: `error("some
description", {"url": url}, cause=e)`
2. I prevent exception leakage; I have no idea the diversity of
exceptions my code can raise, so I CNCR.
3. Every exception is it's own unique type; I can switch on the message
if I want (I rarely do this, but it happens, see below)
Can Python provide better support for the CNCR pattern? If it is
lightweight enough, maybe people will use it, and then we can say
something useful about the (restricted) range of exceptions coming from
a method:
A context manager, a `with` block, could do this:
| def my_method():
| with Explanation("some description"):
| # do something
I propose a simple line, which effectively defines a try block to the
end of the current code block. Call it the `on-raises` syntax:
| def my_method():
| on Exception raises "some description"
| # do something
It is better than the `with` block because:
* it does not put code in the happy path
* it has less indentation
* we can conclude "some description" is the only exception raised by
this method
The `on-raises` need not start at the beginning of a code block, but
then we can say less about what exceptions come out of `my_method`:
| def my_method():
| # no exception checks here
| on Exception raises "some description"
| # do something
Since `on-raises` can be used in any code block, we can save some
indentation. Instead of
| def my_method():
| with some_file:
| try:
| # do something
| except Exception as e:
| raise Exception("some description") from e
we have
| def my_method():
| with some_file:
| on Exception raises "some description"
| # do something
of course we can have nested `on-raises`,
| def my_method():
| on Exception raises "bigger problem"
| with some_file:
| on Exception raises "some description"
| # do something
Plus we know only "bigger problem" exceptions can be raised.
The above is the same as:
| def my_method():
| try:
| with some_file:
| try:
| # do something
| except Exception as e:
| raise Exception("some description") from e
| except Exception as e:
| raise Exception("bigger problem") from e
in the rare case we actually want to deal with an exception, we revert
back to trusty old try/except:
| def my_other_method():
| on Exception raises "some other description"
| try:
| my_method()
| except "some description" as e:
| # I know how to handle this case
| return
which is the same as:
| def my_other_method():
| try:
| my_method()
| except Exception as e:
| if "some description" in e:
| # I know how to handle this case
| return
| error("some other description", cause=e)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Thu, Feb 21, 2019 at 11:32 PM Chris Angelico wrote: > On Fri, Feb 22, 2019 at 2:27 PM Juancarlo Añez wrote: > > Then, if exceptions are going to be part of a type, there should be a > way to express the semantics of them (like in Eiffel), so > stack.pop();stack.push(x) doesn't have to catch StackFullException. > > > > That assumes atomicity. If you want an atomic "replace top of stack" > that can never raise StackFullException, it's probably best to express > it as stack.replacetop(x) rather than having something that might be > interrupted. > Ah! What to do with an exception in a concurrent context? Abort, or retry after a while? > People do dumb things with exceptions, yes. Why does this mean that > they are bad? I don't understand this. Exception handling (and stack > unwinding) gives an easy and clear way to refactor code without having > to daisychain error handling everywhere. How is throwing that away > going to help people write better code? > For programs that are recursive or multi-layered, exceptions are a clean and efficient way to unwind. This PEG parser generator I wrote was made possible because Python exceptions are semantically clean and very efficient: https://github.com/neogeny/TatSu/blob/master/tatsu/contexts.py > But then, Golang also decided that Unicode wasn't necessary, and we > should all deal with UTF-8 encoded byte sequences instead of text > strings, so I'm fairly sure there are no ten foot barge poles long > enough for me to touch it with. There are languages that have problems > because of history (*cough*JavaScript*cough*), but for a new language > to make multiple poor decisions just means it's one to avoid. > The quest for a programming language in which _"anyone"_ can program, _without_ making mistakes, has never ended, and probably never will. "Alexa! Please write the software for a new Internet email system that overcomes the limitations of the current one!" Sometimes staying away is not an option. -- Juancarlo *Añez* ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Sat, Feb 23, 2019 at 5:32 AM Juancarlo Añez wrote: > On Thu, Feb 21, 2019 at 11:32 PM Chris Angelico wrote: >> >> On Fri, Feb 22, 2019 at 2:27 PM Juancarlo Añez wrote: >> > Then, if exceptions are going to be part of a type, there should be a way >> > to express the semantics of them (like in Eiffel), so >> > stack.pop();stack.push(x) doesn't have to catch StackFullException. >> > >> >> That assumes atomicity. If you want an atomic "replace top of stack" >> that can never raise StackFullException, it's probably best to express >> it as stack.replacetop(x) rather than having something that might be >> interrupted. > > Ah! What to do with an exception in a concurrent context? Abort, or retry > after a while? You mean in a situation where another thread might be pushing/popping on the same stack? That's up to you. Exception handling works exactly the same way with multiple threads as it does with a single thread; each context has a call stack. Things are a bit more complicated with other forms of concurrency, but at the point where there is an execution context again, that's where exceptions can start bubbling again. That's how generators work, for instance. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Fri, Feb 22, 2019 at 2:27 PM Juancarlo Añez wrote:
> Then, if exceptions are going to be part of a type, there should be a way to
> express the semantics of them (like in Eiffel), so stack.pop();stack.push(x)
> doesn't have to catch StackFullException.
>
That assumes atomicity. If you want an atomic "replace top of stack"
that can never raise StackFullException, it's probably best to express
it as stack.replacetop(x) rather than having something that might be
interrupted.
> Newer languages like Go and Swift shy away from exceptions because of the
> tendency to:
>
> try:
># something
> except:
> print('oops!)
>
People do dumb things with exceptions, yes. Why does this mean that
they are bad? I don't understand this. Exception handling (and stack
unwinding) gives an easy and clear way to refactor code without having
to daisychain error handling everywhere. How is throwing that away
going to help people write better code?
But then, Golang also decided that Unicode wasn't necessary, and we
should all deal with UTF-8 encoded byte sequences instead of text
strings, so I'm fairly sure there are no ten foot barge poles long
enough for me to touch it with. There are languages that have problems
because of history (*cough*JavaScript*cough*), but for a new language
to make multiple poor decisions just means it's one to avoid.
ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Thu, Feb 21, 2019 at 10:34 PM Ben Rudiak-Gould
wrote:
> > It's well documented how checked exceptions lead to bad code.
>
> Please cite a paper. I know "everyone knows" that they're bad, but
> "everyone knows" a lot of things.
>
This is one of the original interviews touching why checked exceptions were
not made part of C#:
https://www.artima.com/intv/handcuffs.html
The problem is that a code fragment should only catch exceptions that it
expects, because it won't know what to do anything else but to wrap the
exception into one of the enclosing type.
But if every type has to define a SomethingWentWrongInTypeException wrapper
exception, it's logical that all of them inherit from a standard
SomethingWentWrongException. Having done that, the type-specific generic
exceptions are of no value, because any code wanting to guard against the
unexpected will just catch SomethingWentWrongException. And that brings us
back to square one, which is letting unexpected exceptions through to
wherever a clean shutdown or restart of the subsystem can be done.
Then, if exceptions are going to be part of a type, there should be a way
to express the semantics of them (like in Eiffel), so
stack.pop();stack.push(x) doesn't have to catch StackFullException.
In the end, I think that allowing type-hints about exceptions that may be
raised is quite useful, as long as they are not checked/enforced (in the
style of Java). Many times I've missed a PyCharm hint me about the actual
exceptions a given call may actually raise.
Newer languages like Go and Swift shy away from exceptions because of the
tendency to:
try:
# something
except:
print('oops!)
--
Juancarlo *Añez*
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
> It's well documented how checked exceptions lead to bad code. Please cite a paper. I know "everyone knows" that they're bad, but "everyone knows" a lot of things. Here's a recentish proposal by Herb Sutter to add a kind of checked exception to C++: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0709r0.pdf He talks about some of the same issues I've talked about in this thread: in particular, that the farther exceptions propagate, the less likely that they can be handled in a precise way. In practice, exceptions are either handled early or they are handled generically (e.g. by printing a traceback). > Exceptions may be logically thought as of part of the type of a method, Yes, exceptions designed to be caught and handled by the caller are essentially alternate return values. Examples are KeyError for mapping lookups, FileNotFoundError for open(), etc. They are part of the method's interface whether they are encoded in a type system or not. > but that requires that the type catches every exception that may be raised > from the implementation and either handles it, or translates it to one > belonging to the type. What you're describing is strong static typing. Yes, it is a hassle. It means you have to do a lot of up-front work proving that your program will handle *every* case before the implementation will permit you to run it on *any* case. A lot of programmers don't like that. They want to write code that works for the cases they care about in the moment, and think about the other cases "later" (which in practice often means "after the product has shipped and someone files a bug report"). > It's a lot of work, it's cumbersome, and it is fragile, as the exception > handling may need to change over minor implementation changes. Yes, it's a lot of work and cumbersome. In exchange for this extra effort, you get static guarantees that make it easier to reason about the behavior of the program. > The strategy of catching only exceptions of interest and letting others pass > produces less fragile and easier to test code. It is less of a hassle to write unchecked code. I don't know if it's easier to test, but maybe. It isn't less fragile, at least not in the way I understand "fragile." Fragile code isn't code that is prone to compile-time type errors when it's changed. Fragile code is code that's prone to crashes or other broken behavior at runtime when it's changed, because of hidden constraints that weren't expressible in the type system. At least, that's what "fragile" means in "fragile base class problem." ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Thu, Feb 21, 2019 at 6:28 PM Ivan Levkivskyi wrote: > The idea about "checked exceptions" appeared several times in various > places. I used to think that this would be hard to support for any > realistic use cases. But now I think there may be a chance > to turn this into a usable feature if one frames it correctly (e.g. the > focus could be on user defined exceptions, rather than on standard ones, > since there is no way we can annotate every function in typeshed). > It's well documented how checked exceptions lead to bad code. That's why C#, which came after Java, didn't include them. Exceptions may be logically thought as of part of the type of a method, and of the type of the method's class, but that requires that the type catches every exception that may be raised from the implementation and either handles it, or translates it to one belonging to the type. It's a lot of work, it's cumbersome, and it is fragile, as the exception handling may need to change over minor implementation changes. If dependencies don't treat the exceptions that may escape from them as part of the type, then our own types will need to be changes every time a dependency changes its implementation. The strategy of catching only exceptions of interest and letting others pass produces less fragile and easier to test code. -- Juancarlo *Añez* ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Tue, 19 Feb 2019 at 21:07, Miikka Salminen wrote: > Hi! > > To help automatic document generators and static type checkers reason more > about the code, the possible side-effects and raised exceptions could also > be annotated in a standardized way. > The idea about "checked exceptions" appeared several times in various places. I used to think that this would be hard to support for any realistic use cases. But now I think there may be a chance to turn this into a usable feature if one frames it correctly (e.g. the focus could be on user defined exceptions, rather than on standard ones, since there is no way we can annotate every function in typeshed). But this is probably not the best place to the start the discussion (we may come back here if we will have a proposal). I would recommend to post either on typing GitHub tracker, or mypy GitHub tracker, or on typing SIG mailing list. IMO the part about side effects is a non-starter. -- Ivan ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Thu, Feb 21, 2019 at 10:04 PM Steven D'Aprano wrote: > > On Thu, Feb 21, 2019 at 07:05:55PM +1100, Chris Angelico wrote: > > [Ben] > > > Other functions also conceptually have three ways of returning: > > > ordinary return with a value, a documented special return like > > > KeyError, and pass-through exceptions. If the pass-through exception > > > is KeyError, it gets conflated with the documented exceptional return, > > > but correct code should handle them differently. It doesn't matter > > > whether the syntax for the documented special return is "return x" or > > > "raise KeyError(x)". > > > > Not sure what you mean here. If the documented special return is > > "return x", then it's a value that's being returned. That's completely > > different from raising some sort of exception. > > I think I understand what Ben means, because I think I've experimented > with functions which do what he may be getting at. > > Remember that exceptions are not necessarily errors. So you might write > a function which returns a value in the standard case, but raises an > exception to represent an exceptional case. If this is not an error, > then the caller ought to be prepared for the exception and always catch > it. Yep, I understand that part (and mention KeyError specifically). What I don't understand is the documented special return of "return x". If there's a way to return a magic value, then it's still just a value. For the rest, yeah, there's the normal Python behaviour of signalling "nope" by raising a specific exception. And yes, leaking an exception of the same type from within that function is going to be interpreted as that "nope". That's important to the ability to refactor - you can have a helper function that raises, and then the main __getitem__ or __getattr__ or whatever will just propagate the exception. That's why "leaking" is such a hard thing to pin down. ChrisA ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Thu, Feb 21, 2019 at 07:05:55PM +1100, Chris Angelico wrote: [Ben] > > Other functions also conceptually have three ways of returning: > > ordinary return with a value, a documented special return like > > KeyError, and pass-through exceptions. If the pass-through exception > > is KeyError, it gets conflated with the documented exceptional return, > > but correct code should handle them differently. It doesn't matter > > whether the syntax for the documented special return is "return x" or > > "raise KeyError(x)". > > Not sure what you mean here. If the documented special return is > "return x", then it's a value that's being returned. That's completely > different from raising some sort of exception. I think I understand what Ben means, because I think I've experimented with functions which do what he may be getting at. Remember that exceptions are not necessarily errors. So you might write a function which returns a value in the standard case, but raises an exception to represent an exceptional case. If this is not an error, then the caller ought to be prepared for the exception and always catch it. This could be considered an alternative design to returning a tuple: (True, value) # standard case (False,) # or raise an exception If this seems strange, it actually isn't *that* strange. It is very similar to the way iterators yield a value in the standard case, and raise StopIteration to signal the non-standard, non-erroneous but exception case of having reached the end of the iterator. Another potential example would be searching, where "Not Found" is not necessarily an error, but it is always an exceptional case. The built-ins raise KeyError or ValueError (for str.index) but they could have just as easily raise KeyNotFound and SubstringNotFound. So exceptions do not necessarily represent errors that should bubble up to the user, to be reported as a bug. They can also represent an alternative (if slightly clumsy) mechanism for passing information to the caller. -- Steven ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Wed, Feb 20, 2019 at 11:52 PM Ben Rudiak-Gould wrote: > Other functions also conceptually have three ways of returning: > ordinary return with a value, a documented special return like > KeyError, and pass-through exceptions. well, I wouldn't call that three ways of returning... But yes, there is no (easy) way to distinguish an Exception raised by the function you called, and one raised somewhere deeper that. And I have been bitten by that more than once. It makes "Easier to ask forgiveness than permission" kind of tricky. But I've found that good unit tests help a lot. And Exception handling is messy -- the point made by the OP, I'm not sure there's a better way to do it. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Thu, Feb 21, 2019 at 6:51 PM Ben Rudiak-Gould wrote:
>
> On Wed, Feb 20, 2019 at 2:43 AM Chris Angelico wrote:
> > That's because a generator function conceptually has three ways to
> > provide data (yield, return, and raise), but mechanically, one of them
> > is implemented over the other ("return" is "raise StopIteration with a
> > value"). For other raised exceptions, this isn't a problem.
>
> Other functions also conceptually have three ways of returning:
> ordinary return with a value, a documented special return like
> KeyError, and pass-through exceptions. If the pass-through exception
> is KeyError, it gets conflated with the documented exceptional return,
> but correct code should handle them differently. It doesn't matter
> whether the syntax for the documented special return is "return x" or
> "raise KeyError(x)".
Not sure what you mean here. If the documented special return is
"return x", then it's a value that's being returned. That's completely
different from raising some sort of exception.
You have a reasonable point about raising KeyError. It's hard to catch
bugs inside __getitem__ that result in the leakage of a KeyError. But
it's not uncommon to implement getitem over some other object's
getitem, which means that leakage is absolutely correct. I'm not sure
what could really be done about that, but I'm also not sure it's
necessary; those special methods tend to be very short. If you're
writing a large and complex __getitem__, it might be worth adding some
overhead to help with potential debugging, maybe wrapping most of your
logic in a generator and having "yield" become "return", "return"
become "raise KeyError", and "raise KeyError" become "raise
RuntimeError". Easy enough with a decorator if you want it.
Unnecessary overhead for most classes though.
ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Wed, Feb 20, 2019 at 2:43 AM Chris Angelico wrote:
> That's because a generator function conceptually has three ways to
> provide data (yield, return, and raise), but mechanically, one of them
> is implemented over the other ("return" is "raise StopIteration with a
> value"). For other raised exceptions, this isn't a problem.
Other functions also conceptually have three ways of returning:
ordinary return with a value, a documented special return like
KeyError, and pass-through exceptions. If the pass-through exception
is KeyError, it gets conflated with the documented exceptional return,
but correct code should handle them differently. It doesn't matter
whether the syntax for the documented special return is "return x" or
"raise KeyError(x)". I've never been screwed by this as badly with
other exceptions as I was by StopIteration, but it's a general problem
with the design of exceptions.
I don't think exception specifications would solve that problem since
you probably couldn't describe the KeyError's origin in the spec
either. But that doesn't mean it isn't a problem.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Wed, Feb 20, 2019 at 9:09 PM Ben Rudiak-Gould wrote:
> That problem, of an inadvertently leaked implementation detail
> masquerading as a proper alternate return value, used to be a huge
> issue with StopIteration, causing bugs that were very hard to track
> down, until PEP 479 fixed it by translating StopIteration into
> RuntimeError when it crossed an abstraction boundary.
That's because a generator function conceptually has three ways to
provide data (yield, return, and raise), but mechanically, one of them
is implemented over the other ("return" is "raise StopIteration with a
value"). For other raised exceptions, this isn't a problem.
> I think converting exceptions to RuntimeErrors (keeping all original
> information, but bypassing catch blocks intended for specific
> exceptions) is the best option. (Well, second best after ML/Haskell.)
> But to make it work you probably need to support some sort of
> exception specification.
The trouble with that is that it makes refactoring very hard. You
can't create a helper function without knowing exactly what it might
be raising.
> I'm rambling. I suppose my points are:
>
> * Error handing is inherently hard, and all approaches do badly
> because it's hard and programmers hate it.
Well, yeah, no kidding. :)
> ... I was bitten several times by
> that StopIteration problem.
>
There's often a completely different approach that doesn't leak
StopIteration. One of my workmates started seeing RuntimeErrors, and
figured he'd need a try/except in this code:
def _generator(self, left, right):
while True:
yield self.operator(next(left), next(right))
But instead of messing with try/except, it's much simpler to use
something else - in this case, zip.
Generally, it's better to keep things simple rather than to complicate
them with new boundaries. Unless there's a good reason to prevent
leakage, I would just let exception handling do whatever it wants to.
But if you want to specifically say "this function will not raise
anything other than these specific exceptions", that can be done with
a decorator:
def raises(*exc):
"""Declare and enforce what a function will raise
The decorated function will not raise any Exception other than
the specified ones, or RuntimeError.
"""
def deco(func):
@functools.wraps(func)
def convert_exc(*a, **kw):
try: return func(*a, **kw)
except exc: raise
except Exception as e: raise RuntimeError from e
convert_exc.may_raise = exc # in case it's wanted
return convert_exc
return deco
This way, undecorated functions behave as normal, so refactoring isn't
impacted. But if you want the help of a declared list of exceptions,
you can have it.
@raises(ValueError, IndexError)
def frob(x):
...
ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Tue, Feb 19, 2019 at 3:19 PM Steven D'Aprano wrote:
> And I note that in Java, where the idea of checked exceptions
> originated, it turned out to be full of problems and a very bad idea.
What should be the default behavior of a language when the programmer
doesn't explicitly handle an error? Options include:
1. Type error (Java, ML/Haskell)
2. Ignore it (C)
3. Pass it as-is to the caller
4. "Genericize" it, e.g. wrap it in a RuntimeError, then pass to the caller
The problem with the Java approach is that people don't want to think
about how to properly handle every error, and just wrap their code in
catch (...) {} instead. I think it works much better in ML/Haskell,
though perhaps only because the average skill level of the programmers
is higher.
The problem with the C approach is that people don't want to think
about how to properly handle every error, and just call every function
in a void context.
The problem with passing exceptions as-is to the caller is that
they're very often implementation details. If you're lucky, they will
propagate to a generic catch-all somewhere which will generate a
traceback that a human may be able to use to fix the problem. If
you're unlucky, the caller wrote `return d[k].frobnicate()` inside a
try block and frobnicate's internal KeyError gets misinterpreted as a
lookup failure in d.
That problem, of an inadvertently leaked implementation detail
masquerading as a proper alternate return value, used to be a huge
issue with StopIteration, causing bugs that were very hard to track
down, until PEP 479 fixed it by translating StopIteration into
RuntimeError when it crossed an abstraction boundary.
I think converting exceptions to RuntimeErrors (keeping all original
information, but bypassing catch blocks intended for specific
exceptions) is the best option. (Well, second best after ML/Haskell.)
But to make it work you probably need to support some sort of
exception specification.
I'm rambling. I suppose my points are:
* Error handing is inherently hard, and all approaches do badly
because it's hard and programmers hate it.
* Stronger typing is only bad if you just want your code to run and
are tired of fixing it to be type-correct. The people who voluntarily
add type annotations to Python programs probably aren't those kinds of
people; they're probably much more likely than the average programmer
to want checked exceptions.
* Declaring that a function only raises Foo doesn't have to mean "it
raises Foo, and also passes exceptions from subfunctions to the caller
unchanged, no matter what they are." It could also mean "it raises
Foo, and converts other exceptions into RuntimeError." This would
actually be useful because it would mean that you could safely put
longer expressions in try blocks, instead of defensively just putting
the one method call whose KeyError you want to handle in the try:
block, and moving everything else to the else: block, as I tend to do
all the time because I'm paranoid and I was bitten several times by
that StopIteration problem.
-- Ben
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
Chris Angelico wrote: if I'd been able to ascertain which functions were pure, or at least side-effect-free, it would have saved me a lot of hassle.) I'm not sure how useful such annotations would be in practice, though. They're only as good as the diligence of the author in using them consistently and correctly, which makes them not much better than comments. -- Greg ___ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Wed, Feb 20, 2019 at 10:18 AM Steven D'Aprano wrote:
> So the promises made will nearly always be incredibly weak:
>
> - raises(A) means "may raise A, or any other unexpected exception"
> - sideeffects(True) means "may have expected side-effects"
> - sideeffects(False) means "may have unexpected side-effects"
Perhaps, but sometimes it's nice to know that a function is not
intended to have any side effects. For instance, I'm currently working
on some code that has a bunch of lines like this:
type = _decode("WeaponTypes" if is_weapon else "ItemTypes")
balance = _decode("BalanceDefs")
brand = _decode("Manufacturers")
Suppose we decide that the "balance" variable isn't being used
anywhere. Is it okay to not decode the BalanceDefs? Nope, it isn't,
because the decoding is done sequentially, which means that omitting
one will leave all the subsequent ones desynchronized. Knowing that a
function has no intentional side effects (which is weaker than
declaring that it's a pure function) may well be useful when
refactoring a messy codebase.
(A messy codebase, such as the one that I was reading through when
building that code. It's not code I'm intrinsically proud of, but I
can say with confidence that it is WAY cleaner than the reference
code. And if I'd been able to ascertain which functions were pure, or
at least side-effect-free, it would have saved me a lot of hassle.)
ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
On Tue, Feb 19, 2019 at 11:05:55PM +0200, Miikka Salminen wrote:
> Hi!
>
> To help automatic document generators and static type checkers reason more
> about the code, the possible side-effects and raised exceptions could also
> be annotated in a standardized way.
This is Python. Nearly everything could have side-effects or raise
exceptions *wink*
Despite the wink, I am actually serious. I fear that this is not
compatible with duck-typing. Take this simple example:
@raises(TypeError)
def maximum(values:Iterable)->Any:
it = iter(values)
try:
biggest = next(it)
except StopIteration:
return None
for x in it:
if x > biggest:
biggest = x
return x
But in fact this code can raise anything, or have side-effects, since it
calls x.__gt__ and that method could do anything.
So our decoration about raising TypeError is misleading if you read it
as "this is the only exception the function can raise". You should read
it as "this promises to sometimes raise TypeError, but could raise any
other exception as well".
This pretty much shows that the idea of checked exceptions doesn't go
well with Python's duck-typing.
And I note that in Java, where the idea of checked exceptions
originated, it turned out to be full of problems and a very bad idea.
Here is your example:
> In [3]: @raises(ValueError)
>...: def hello_if_5(x: int) -> None:
>...: if x != 5:
>...: raise ValueError("number other than 5 given")
>...: print("Hello!")
In this *trivial* function, we can reason that there are no other
possible exceptions (unless ValueError or print are monkey-patched or
shadowed). But how many of your functions are really that simple? I
would expect very few.
For the vast majority of cases, any time you decorate a non-trivial
function with "raises(X)", it needs to be read as "can raise X, or any
other exception".
And similarly with annotating functions for side-effects. I expect that
most functions need to be read as "may have side-effects" even if
annotated as side-effect free.
So the promises made will nearly always be incredibly weak:
- raises(A) means "may raise A, or any other unexpected exception"
- sideeffects(True) means "may have expected side-effects"
- sideeffects(False) means "may have unexpected side-effects"
I don't think there is much value in a static checker trying to reason
about either. (And the experience of Java tells us that checking
exceptions is a bad idea even when enforced by the compiler.) If I'm
right, then adding support for this to the std lib is unnecessary.
But I could be wrong, and I encourage static checkers to experiment. To
do so, they don't need support from the std lib. They can provide their
own decorator, or use a comment:
# raises ValueError, TypeError
# +sideeffects
def spam():
...
--
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Type hints for functions with side-effects and for functions raising exceptions
Hi!
To help automatic document generators and static type checkers reason more
about the code, the possible side-effects and raised exceptions could also
be annotated in a standardized way.
I'll let some example code talk on my behalf. Here's a simple decorator for
annotating exceptions:
In [1]: import typing as t
In [2]: def raises(exc: Exception) -> t.Callable:
...: def decorator(fn: t.Callable) -> t.Callable:
...: fn.__annotations__["__raises__"] = exc
...: return fn
...: return decorator
...:
In [3]: @raises(ValueError)
...: def hello_if_5(x: int) -> None:
...: if x != 5:
...: raise ValueError("number other than 5 given")
...: print("Hello!")
...:
In [4]: hello_if_5.__annotations__
Out[4]: {'x': int, 'return': None, '__raises__': ValueError}
In [5]: hello_if_5(1)
---
ValueErrorTraceback (most recent call last)
in
> 1 hello_if_5(1)
in hello_if_5(x)
2 def hello_if_5(x: int) -> None:
3 if x != 5:
> 4 raise ValueError("number other than 5 given")
5 print("Hello!")
6
ValueError: number other than 5 given
In [6]: hello_if_5(5)
Hello!
In [7]:
and here's a simple decorator for annotating side-effects:
In [1]: import typing as t
In [2]: def side_effect(has_side_effect: bool) -> t.Callable:
...: def decorator(fn: t.Callable) -> t.Callable:
...: fn.__annotations__["__side_effect__"] = has_side_effect
...: return fn
...: return decorator
...:
In [3]: a = 10
In [4]: @side_effect(True)
...: def change_a(val: int) -> None:
...: global a
...: a = val
...:
In [5]: change_a.__annotations__
Out[5]: {'val': int, 'return': None, '__side_effect__': True}
In [6]: change_a(100)
In [7]: a
Out[7]: 100
In [8]: @side_effect(True)
...: def mutate_list(items: t.List) -> None:
...: items.append("new item")
...:
In [9]: mutate_list.__annotations__
Out[9]: {'items': typing.List, 'return': None, '__side_effect__': True}
In [10]: l = ["old item"]
In [11]: mutate_list(l)
In [12]: l
Out[12]: ['old item', 'new item']
In [13]:
The example implementations have some obvious limitations, such as only
allowing one error type. What do you think of the idea in general? Do you
feel this is something that could be included in Python? typing would
probably be a good module to store such decorators, I guess…
Miikka Salminen
miikka.salmi...@gmail.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
