Chris already responded; I'll top-post to clarify my position.
I acted quickly because too many people were expending too much energy
debating the issue without coming up with a different resolution, and I am
unhappy with the status quo.
Many people don't seem to see the difference between the iterator *protocol*,
which determines how the caller discovered that the iterator is exhausted,
and various iterator *implementations*, which determine how you feed a
series of values to the caller and indicate the end of that series.
Generators are one possible implementation.
The crucial insight, leading to the solution, is that a generator is one
step removed from the object representing the protocol. When you call a
generator, two objects are created: a generator protocol object (this is
what we typically refer to as a generator object, as opposed to a generator
function), and a stack frame. The stack frame is initially in a suspended
state, and is poised to execute the code of the generator function.
The protocol object has a __next__() method. When this is called, the
associated stack frame is resumed until it either yields or returns -- if
it yields, __next__() returns a value, but if it returns, __next__() raises
StopIteration. There is no need for the stack frame to raise StopIteration
-- the StopIteration is provided "for free" by the generator protocol
object.
The confusion stems from the fact that if the stack frame neither yields
nor returns but raises an exception, this exception is simply passed
through the protocol object, and in most cases (e.g. an AttributeError)
this simply bubbles out until it is either caught or terminates the program
or thread with a printed stack trace. But in the special case that the
frame raises StopIteration, once this is raised by the protocol object's
__next__(), this is indistinguishable from the StopIteration that this
object throws when the stack frame simply returns.
It wouldn't be so bad if we had the occasional generator author writing
"raise StopIteration" instead of "return" to exit from a generator. (We
could just add a recommendation against this to the style guide.) But the
problem is that an unguarded next() call also raises StopIteration.
Sometimes this is intentional (as in some itertools examples). But
sometimes an unguarded next() call occurs deep in the bowels of some code
called by the generator, and this situation is often hard to debug, since
there is no stack track.
One more thing. You correctly say that the built-in next() function is not
the same thing as the __next__() method of the protocol. While this is
true, tinkering with next() is not feasible -- if we were to change the
interface of next() we would have to change pretty much every call to it.
(I did a little research on a large proprietary codebase to which I have
access -- I found innumerable next() calls, almost all of them guarded by a
"try: ... except StopIteration: ..." block. Even the ones inside
generators. On the other hand, I found almost no code depending on the
behavior that the PEP changes, apart from a few places where a redundant
"raise StopIteration" was found at the end of a generator.
So. To reiterate my point. A generator frame uses "yield <value>" to
produce a value and "return" to terminate the iteration; the generator
protocol object wrapping the frame translates these into "return <value>"
and "raise StopIteration", respectively. Raising StopIteration in the frame
is redundant and can cause errors to pass silently, hence this is the right
thing to change.
On Sun, Nov 23, 2014 at 12:18 PM, Mark Shannon <m...@hotpy.org> wrote:
> Hi,
>
> I have serious concerns about this PEP, and would ask you to reconsider it.
>
> [ Very short summary:
> Generators are not the problem. It is the naive use of next() in an
> iterator that is the problem. (Note that all the examples involve calls to
> next()).
> Change next() rather than fiddling with generators.
> ]
>
> I have five main concerns with PEP 479.
> 1. Is the problem, as stated by the PEP, really the problem at all?
> 2. The proposed solution does not address the underlying problem.
> 3. It breaks a fundamental aspect of generators, that they are iterators.
> 4. This will be a hindrance to porting code from Python 2 to Python 3.
> 5. The behaviour of next() is not considered, even though it is the real
> cause of the problem (if there is a problem).
>
> 1. The PEP states that "The interaction of generators and StopIteration is
> currently somewhat surprising, and can conceal obscure bugs."
> I don't believe that to be the case; if someone knows what StopIteration
> is and how it is used, then the interaction is entirely as expected.
>
> I believe the naive use of next() in an iterator to be the underlying
> problem.
> The interaction of generators and next() is just a special case of this.
>
> StopIteration is not a normal exception, indicating a problem, rather it
> exists to signal exhaustion of an iterator.
> However, next() raises StopIteration for an exhausted iterator, which
> really is an error.
> Any iterator code (generator or __next__ method) that calls next() treats
> the StopIteration as a normal exception and propogates it.
> The controlling loop then interprets StopIteration as a signal to stop and
> thus stops.
> *The problem is the implicit shift from signal to error and back to
> signal.*
>
> 2. The proposed solution does not address this issue at all, but rather
> legislates against generators raising StopIteration.
>
> 3. Generators and the iterator protocol were introduced in Python 2.2, 13
> years ago.
> For all of that time the iterator protocol has been defined by the
> __iter__(), next()/__next__() methods and the use of StopIteration to
> terminate iteration.
>
> Generators are a way to write iterators without the clunkiness of explicit
> __iter__() and next()/__next__() methods, but have always obeyed the same
> protocol as all other iterators. This has allowed code to rewritten from
> one form to the other whenever desired.
>
> Do not forget that despite the addition of the send() and throw() methods
> and their secondary role as coroutines, generators have primarily always
> been a clean and elegant way of writing iterators.
>
> 4. Porting from Python 2 to Python 3 seems to be hard enough already.
>
> 5. I think I've already covered this in the other points, but to reiterate
> (excuse the pun):
> Calling next() on an exhausted iterator is, I would suggest, a logical
> error.
> However, next() raises StopIteration which is really a signal to the
> controlling loop.
> The fault is with next() raising StopIteration.
> Generators raising StopIteration is not the problem.
>
> It also worth noting that calling next() is the only place a StopIteration
> exception is likely to occur outside of the iterator protocol.
>
> An example
> ----------
>
> Consider a function to return the value from a set with a single member.
> def value_from_singleton(s):
> if len(s) < 2: #Intentional error here (should be len(s) == 1)
> return next(iter(s))
> raise ValueError("Not a singleton")
>
> Now suppose we pass an empty set to value_from_singleton(s), then we get a
> StopIteration exception, which is a bit weird, but not too bad.
>
> However it is when we use it in a generator (or in the __next__ method of
> an iterator) that we get a serious problem.
> Currently the iterator appears to be exhausted early, which is wrong.
> However, with the proposed change we get RuntimeError("generator raised
> StopIteration") raised, which is also wrong, just in a different way.
>
> Solutions
> ---------
> My preferred "solution" is to do nothing except improving the
> documentation of next(). Explain that it can raise StopIteration which, if
> allowed to propogate can cause premature exhaustion of an iterator.
>
> If something must be done then I would suggest changing the behaviour of
> next() for an exhausted iterator.
> Rather than raise StopIteration it should raise ValueError (or
> IndexError?).
>
> Also, it might be worth considering making StopIteration inherit from
> BaseException, rather than Exception.
>
>
> Cheers,
> Mark.
>
> P.S. 5 days seems a rather short time to respond to a PEP.
> Could we make it at least a couple of weeks in the future,
> or better still specify a closing date for comments.
>
>
>
--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
