[issue25014] Add contextlib.itercm()

[Ezio Melotti]

Ezio Melotti added the comment:

> Having spent a few days pondering this after Ezio first mentioned the
> concept to me on IRC, I'm rejecting this on the basis of "not every 3 
> line function needs to be in the standard library".

When I first mentioned this to Nick on IRC, the implementation of itercm() was 
a not-so-trivial function that called __enter__/__exit__ manually while 
catching StopIteration.  It only occurred to me while posting this issue, that 
the same could be achieved with a simple `yield from` in a `with`.
I also didn't realize that the __exit__ called in case of error in the attached 
example was triggered by the garbage collector.
I therefore agree that a somewhat obscure and non-deterministic three-liner 
doesn't belong in the standard library.  Thanks everyone for the feedback!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Antoine Pitrou]

Antoine Pitrou added the comment:

I don't think I like this idea. It's not really a common use case (I've never 
wished I had itercm()) and it will make it possible to write slightly obscure 
code. Of course you can already write obscure code using itertools :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Raymond Hettinger]

Raymond Hettinger added the comment:

> I don't think I like this idea. It's not really a common use
> case (I've never wished I had itercm()) and it will make it 
> possible to write slightly obscure code. Of course you can
>  already write obscure code using itertools :-)

I concur with Antoine and think the world is better off without this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Nick Coghlan]

Nick Coghlan added the comment:

__exit__ will still get invoked even if the generator is never exhausted:

>>> def itercm(cm):
... with cm:
... yield from cm
... 
>>> class TestCM:
... def __iter__(self):
... yield 6
... yield 9
... yield 42
... def __enter__(self):
... return self
... def __exit__(self, *args):
... print("Terminated CM")
... 
>>> itr = itercm(TestCM())
>>> next(itr)
6
>>> del itr
Terminated CM

We addressed the major problems with generators failing to clean up resources 
back when generator.close() was introduced in PEP 342, and then Antoine 
addressed the cyclic GC problem in PEP 442.

The key thing that itercm() adds over the status quo is that if the generator 
*is* exhausted, then the resource *will* be cleaned up immediately. If the 
generator *isn't* exhausted, then it falls back to non-deterministic GC based 
cleanup, which is what you'd get today by not using a context manager at all.

To be convinced that we need a third cleanup option beyond "always 
deterministic" and "always non-deterministic", I'd need some concrete use cases 
where the success case needs deterministic cleanup, but the error case is OK 
with non-deterministic cleanup.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Martin Panter]

Martin Panter added the comment:

I had also thought of this kind of function, but I dismissed it because it 
would either have to rely on garbage collection or an explicit close() call to 
close the generator in case the iteration is aborted. I think it might need 
some kind of “with-for” combined statement added to the langauge to be 
bulletproof.

Considering the second example in your script, “exit is called in case of 
errors”: What is stopping the interpreter from storing the iterator of the 
current “for” loop in the top-level frame object? Then the iterator would be 
referenced by the exception traceback, and prevent garbage collection of its 
itercm() instance. Hypothetically:

__traceback__ → tb_frame → “for” iterator → itercm() instance

Also, I would tend to put this sort of function in “itertools”, since 
generators are not context managers by design.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Ezio Melotti]

Ezio Melotti added the comment:

FTR one of the reason that led me to itercm() is:

with open(fname) as f:
transformed = (transform(line) for line in f)
filtered = (line for line in lines if filter(line))
# ...

Now filtered must be completely consumed before leaving the body of the `with` 
otherwise this happens:

>>> with open(fname) as f:
... transformed = (transform(line) for line in f)
... filtered = (line for line in lines if filter(line))
... 
>>> # ...
>>> next(filtered)
ValueError: I/O operation on closed file.

With itercm() it's possible to do:

f = itercm(open(fname))
transformed = (transform(line) for line in f)
filtered = (line for line in lines if filter(line))
...
# someone consumes filtered down the line lazily
# and eventually the file gets closed

itercm() could also be used (abused?) where a regular `with` would do just fine 
to save one extra line and indentation level (at the cost of an extra import), 
e.g.:

def lazy_cat(fnames):
for fname in fnames:
yield from itercm(open(fname))

instead of: 

def lazy_cat(fnames):
for fname in fnames:
with open(fname) as f:
yield from f

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Ezio Melotti]

New submission from Ezio Melotti:

Add an itercm() function that receives an iterable that supports the context 
manager protocol (e.g. files) and calls enter/exit without having to use the 
with statement explicitly.

The implementation is pretty straightforward (unless I'm missing something):

def itercm(cm):
with cm:
yield from cm

Example usages:

def cat(fnames):
lines = chain.from_iterable(itercm(open(f)) for f in fnames)
for line in lines:
print(line, end='')

This will close the files as soon as the last line is read.

The __exit__ won't be called until the generator is exhausted, so the user 
should make sure that it is (if he wants __exit__ to be closed).  __exit__ is 
still called in case of exception.

Attached a clearer example of how it works.

Do you think this would be a good addition to contextlib (or perhaps itertools)?


P.S. I'm also contemplating the idea of having e.g. it = itercm(fname, 
func=open) to call func lazily once the first next(it) happens, but I haven't 
thought in detail about the implications of this.  I also haven't considered 
how this interacts with coroutines.

--
components: Library (Lib)
files: itercm-example.txt
messages: 249991
nosy: ezio.melotti, ncoghlan, rhettinger
priority: normal
severity: normal
stage: needs patch
status: open
title: Add contextlib.itercm()
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file40380/itercm-example.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

What if the last line will be never read? We had bugs with resource leaks in 
generators, and I'm not sure that all such bugs were fixed.

--
nosy: +pitrou, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25014] Add contextlib.itercm()

[Ezio Melotti]

Ezio Melotti added the comment:

If you are talking about generators that never get exhausted, the fact that the 
__exit__ is never invoked is expected and something that developers should take 
into account while using itercm().
I'm not aware of other generators-related issues that might cause leaks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com