On 10/19/2016 12:38 AM, Nathaniel Smith wrote:
I'd like to propose that Python's iterator protocol be enhanced to add
a first-class notion of completion / cleanup.
With respect the the standard iterator protocol, a very solid -1 from
me. (I leave commenting specifically on __aiterclose__ to Yury.)
1. I consider the introduction of iterables and the new iterator
protocol in 2.2 and their gradual replacement of lists in many
situations to be the greatest enhancement to Python since 1.3 (my first
version). They are, to me, they one of Python's greatest features and
the minimal nature of the protocol an essential part of what makes them
great.
2. I think you greatly underestimate the negative impact, just as we did
with changing str is bytes to str is unicode. The change itself,
embodied in for loops, will break most non-trivial programs. You
yourself note that there will have to be pervasive changes in the stdlib
just to begin fixing the breakage.
3. Though perhaps common for what you do, the need for the change is
extremely rare in the overall Python world. Iterators depending on an
external resource are rare (< 1%, I would think). Incomplete iteration
is also rare (also < 1%, I think). And resources do not always need to
releases immediately.
4. Previous proposals to officially augment the iterator protocol, even
with optional methods, have been rejected, and I think this one should
be too.
a. Add .__len__ as an option. We added __length_hint__, which an
iterator may implement, but which is not part of the iterator protocol.
It is also ignored by bool().
b., c. Add __bool__ and/or peek(). I posted a LookAhead wrapper class
that implements both for most any iterable. I suspect that the is
rarely used.
def read_newline_separated_json(path):
with open(path) as file_handle: # <-- with block
for line in file_handle:
yield json.loads(line)
One problem with passing paths around is that it makes the receiving
function hard to test. I think functions should at least optionally
take an iterable of lines, and make the open part optional. But then
closing should also be conditional.
If the combination of 'with', 'for', and 'yield' do not work together,
then do something else, rather than changing the meaning of 'for'.
Moving responsibility for closing the file from 'with' to 'for', makes
'with' pretty useless, while overloading 'for' with something that is
rarely needed. This does not strike me as the right solution to the
problem.
for document in read_newline_separated_json(path): # <-- outer for loop
...
If the outer loop determines when the file should be closed, then why
not open it there? What fails with
try:
lines = open(path)
gen = read_newline_separated_json(lines)
for doc in gen: do_something(doc)
finally:
lines.close
# and/or gen.throw(...) to stop the generator.
--
Terry Jan Reedy
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
