[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson  added the comment:

Having given this some thougt, years laters, I believe it _is_ possible to 
write nested() (and nested_delayed()) in a correct way in python, without the 
ContextManagerExit function.

Behold!

import contextlib


@contextlib.contextmanager
def nested_delayed(*callables):
"""
Instantiate and invoke context managers in a nested way.  each argument 
is a callable which
returns an instantiated context manager
"""
if len(callables) > 1:
with nested_delayed(*callables[:-1]) as a, callables[-1]() as b:
yield a + (b,)
elif len(callables) == 1:
with callables[0]() as a:
yield (a,)
else:
yield ()


def nested(*managers):
"""
Invoke preinstantiated context managers in a nested way
"""
def helper(m):
"""
A helper that returns the preinstantiated context manager when 
invoked
"""
def callable():
return m
return callable
return nested_delayed(*(helper(m) for m in managers))



@contextlib.contextmanager
def ca():
print("a")
yield 1

class cb:
def __init__(self):
print ("instantiating b")
def __enter__(self):
print ("b")
return 2
def __exit__(*args):
pass

@contextlib.contextmanager
def cc():
print("c")
yield 3


combo = nested(ca(), cb(), cc())
combo2 = nested_delayed(ca, cb, cc)

with combo as a:
print("nested", a)

with combo2 as a:
print("nested_delayed", a)

with ca() as a, cb() as b, cc() as c:
print ("syntax", (a, b, c))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson  added the comment:

Great throwback.

As far as I know, context managers are still not first class citizens.  You 
cannot _compose_ two context managers into a new one programmatically in the 
language, in the same way that you can, for instance, compose two functions.  
Not even using "eval()" is this possible.  

This means that the choice of context manager, or context managers, to be used, 
has to be known when writing the program.  You cannot pass an assembled context 
manager in as an argument, or otherwise use a "dynamic" context manager at run 
time, unless you decide to use only a fixed number of nested ones. any 
composition of context managers becomes syntax _at the point of invocation_.

The restriction is similar to not allowing composition of functions, i.e. 
having to write

`fa(fb(fc()))` at the point of invocation and not have the capability of doing
```
def fd():
  return fa(fb(fc))
...
fd()
```

I think my "ContextManagerExit" exception provided an elegant solution to the 
problem and opened up new and exciting possibilities for context managers and 
how to use them.

But this here note is just a lament.  I've stopped contributing to core python 
years ago, because it became more of an excercise in lobbying than anything 
else.
Cheers!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Alex Waygood]

Alex Waygood  added the comment:

Given that this issue has seen no activity for eight years, I am closing it as 
"rejected".

--
nosy: +AlexWaygood
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

locally visible is, I think a very misleading term.  How is

with ignore_error, acquire_resource as r:
   doo_stuff_with_resource(r) #can be silently skipped

any more locally visible than
with acquire_resource_ignore_error as r:
doo_stuff_with resource(r) # can be silently skipped.

? does the nested with syntax immediatelly tell you hey, the body can be 
silently skipped?

Requiring that some context manager patterns must be done with a special syntax 
is odd.  What is more, it prohibits us to abstract away context managers.  For 
instance, you can write a function like this

def execute_with_context(ctxt, fn, args):
with ctxt:
return fn(*args)

but if your context manager is of the kind mentioned, i.e. requiring the double 
syntax, you are screwed.

Basically, what I'm proposing (and what the patch provides) is that you can 
write this code:
@contextmanager
def nestedc(ca, cb):
with ca as a, cb as b:
yield a, b

and have it work for _all_ pair of ca, cb.  This then, allows context managers 
to be used like abstract entities, like other objects in the language.  It is 
_not_ about flow control, but about completeness.

A similar pattern for functions is already possible:
def nestedf(fa, fb):
def helper(v):
return fa(fb(v))
return helper

And so, we could write:
execute_with_context(nestedc(ca, cb), nestedf(fa, fb), ('foo',))

Current python does not allow this for arbitrary pairs ca, cb.  My version 
does.  This is what I'm advocating.  That programmers are given the tool to 
combine context managers if they want.
 

As for contextlib.nested().
I'm not necessarily advocation its resuciation in the standardlib, but adding 
that to the patch here to demonstrate how it now _works_.

Here is a simpler version of contextlib.nested:

@contextmanager
def nested_empty():
yield []

@contextmanager
def nested_append(prev, next):
with prev as a, next as b:
a.append(b)
yield a

def nested(*managers):
total = nested_empty()
for mgr in managers:
total = nested_append(total, mgr)
return total

Pretty nice, no?

Now we come to the argument with nested(open(a), open(b)).
I see your point, but I think that the problem is not due to nested, but to 
open.  Deprecating nested, even as a programming pattern demonstration is 
throwing out the baby with the bathwater.

I´ve coined the term hybrid context manager (at least I think I have)to mean 
resources that are their own context managers.  They're hybrid because they are 
acquired explicitly, but can be released via a context manager.  The context 
manager is a bolt on, an afterthought.  Instead of adding __exit__() to files, 
and allowing
with open(fn) as f: pass
We should have encouraged the use of proper context managers:
with opened(fn) as f: pass
or 
with closing(f):  pass

Now, we unfortunately have files being context managers and widely see the 
pattern
with open(fn) as f, open(fn2) as f2:
pass

But how is this bug here:
with nested(open(fn), open(fn2)) as f, f2: pass

any more devuiys than
f, f2 = open(fn), open(fn2)
with f, f2: pass
?

The problem is that files aren't real context managers but hybrids and this 
is what we should warn people about.  The fact that we do have those hybrids in 
our code base should not be cause to remove tools that are designed to work 
with proper context managers.

The decision to remove nested on these grounds sets the precedence that we 
cannot have any functions that operate on context managers.  In fact, what this 
is really is saying is this:

context managers should only be used with the with statement and only 
instantiated in-line.
Anything else may introduce sublte bugs because some context managers are in 
fact not context managers, but the resource that they manage.


In my opinion, it would have been better to deprecate the use of files as 
context managers, and instead urge people to use proper context managers for 
the:  (the proposed) contextlib.opened and (the existing) contextlib.closing)

K

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Nick Coghlan]

Nick Coghlan added the comment:

I think you make a good case, but I already tried and failed to convince Guido 
of this in PEP 377 (see 
http://www.python.org/dev/peps/pep-0377/#rationale-for-change)

More importantly, see his quoted concerns in 
http://mail.python.org/pipermail/python-dev/2009-March/087263.html

While you have come up with a much simpler *implementation* for PEP 377, which 
imposes no additional overhead in the typical case (unlike my implementation, 
which predated the SETUP_WITH opcode and avoided introducing one, which 
required wrapping every __enter__ call in a separate try/except block), it 
still adds a new builtin exception type, and I thing needs a new builtin 
constant as well.

The latter comes in because I think the bound variable name still needs to be 
set to something, and rather than abusing any existing constant, I think a new 
SkipWith constant for both don't call enter/exit and with statement body was 
skipped would actually be clearer.

I actually think explaining a custom exception and constant is less of a burden 
than explaining why factoring out certain constructs with @contextmanager and 
yield doesn't work properly (that's why I wrote PEP 377 in the first place), 
but Guido is the one that ultimately needs to be convinced of the gain.

--
nosy: +gvanrossum

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Barry A. Warsaw]

Changes by Barry A. Warsaw ba...@python.org:


--
nosy: +barry

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

I've modified the patch.  The problem that nested_delayed was trying to solve 
are hybrid context managers, ones that allocate resources during __init__ and 
release them at exit.  A proper context manager should allocate resources 
during __enter__, and thus a number of them can be created upfront with 
impunity.

Added contextlib.proper to turn a hybrid context manager into a proper one by 
instantiating the hybrid in a delayed fashion.
added contextlib.opened() as a special case that does open() properly.

With this change, and the ability to nest error handling of exceptions stemming 
from __enter__(), nested now works as intended.

--
Added file: http://bugs.python.org/file31195/contextmanagerexit.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Thanks, Eric.
I read that bit and I can't say that I disagree.
And I'm not necessarily advocating that skipping the body become a standard 
feature of context managers.  But it is a necessary functionality if you want 
to be able to dynamically nest one or more context managers, something I think 
Python should be able to do, for completeness, if not only for aesthetic beauty.

Having said that, optionally skipping the body is a far cry from the more 
esoteric constructs achievable with pep 340.

And python _already_ silently skips the body of managed code, if you nest two 
managers:

@contextmanager errordude:
1 // 0
yield
@contextmanager handler:
try:
yield
except ZeroDivisionError:
pass

with handler, errordude:
do_stuff()

These context managers will skip the execution of f.  It will be Python's 
internal decision to do so, of course.  But the with statement already has 
the potential to have the body silently skipped.

What I'm adding here, the ContextManagerExit, is the ability for the context 
manager itself to make the decision, so that the two context managers above can 
be coalesced into one:

with nested(handler, errordude):
do_stuff()

The fact that do_stuff can be silently skipped in the first case, where we 
explicitly have two nested calls, invalidates IMHO the argument that context 
managers should not affect control flow.  why shouldn't it also be skippable in 
the case of a single context manager?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Using my latest patch, the ExitStack inline example can be rewritten:

with ExitStack() as stack:
files = [stack.enter_context(open(fname)) for fname in filenames]
# All opened files will automatically be closed at the end of
# the with statement, even if attempts to open files later
# in the list raise an exception

becomes:
with nested(opened(fname) for fname in filenames) as files:
do_stuff_with_files(files)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Nick Coghlan]

Nick Coghlan added the comment:

Allowing a context manager to skip the statement body isn't a new proposal,
and I previously argued your side. However, with multiple context managers,
there is no invisible flow control. Two context managers are locally
visible, which means the outer one completely encloses the inner one and
can suppress exceptions it throws. Guido explicitly made the decision to
require two managers at the point of use to achieve that behaviour when I
proposed making the change - he doesn't care about allowing a single
context manager to provide that functionality.

For the other question, how does your version of nested keep people from
doing nested(open(fname) for name in names)? That was the core problem
with that style of API: it made it far too easy to introduce a latent
defect when combined with file like objects that eagerly acquire their
resource. It wasn't that it couldn't be used correctly, but that the
natural and obvious way of combining it with open() is silently wrong.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

New submission from Kristján Valur Jónsson:

A proposed patch adds two features to context managers:

1)It has always irked me that it was impossible to assemble nested context 
managers in the python language. See issue #5251.
The main problem, that exceptions in __enter__ cannot be properly handled, is 
fixed by introducing a new core exception, ContextManagerExit.  When raised by 
__enter__(), the body that the context manager protects is skipped.  This 
exception is in the spirit of other semi-internal exceptions such as 
GeneratorExit and StopIteration.  Using this exception, contextlib.nested can 
properly handle the case where the body isn't run because of an internal 
__enter__ exception which is handled by an outer __exit__.

2) The mechanism used in implementing ContextManagerExit above is easily 
extended to allowing a special context manager: None.  This is useful for 
having _optional_ context managers.  E.g. code like this:
with performance_timer():
do_work()

def performance_timer():
if profiling:
return accumulator
return None

None becomes the trivial context manager and its __enter__ and __exit__ calls 
are skipped, along with their overhead.

This patch implements both features.
In addition, it:
1) reintroduces contextlib.nested, which is based on nested_delayed
2) introduces contextlib.nested_delayed, which solves the other problem with 
previous versions of nested, that an inner context manager expression shouldn't 
be evaluated early.  contextlib.nested evaluates callables returning context 
managers, rather than managers directly.
3) Allows contextlib.contextmanager decorated functions to not yield, which 
amounts to skipping the protected body (implicitly raising ContextManagerExit)
4) unittests for the whole thing.

I'll introduce this stuff on python-ideas as well.

--
components: Interpreter Core
files: contextmanagerexit.patch
keywords: patch
messages: 194615
nosy: kristjan.jonsson
priority: normal
severity: normal
status: open
title: Enhanced context managers with ContextManagerExit and None
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file31182/contextmanagerexit.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[R. David Murray]

R. David Murray added the comment:

Your use cases are either already addressed by contextlib.ExitStack, or should 
be addressed in the context of its existence.  It is the replacement for 
contextlib.nested.

--
nosy: +ncoghlan, r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

IMHO, exitstack is not a very nice construct.  It's implementation is far 
longer than contextlib.nested.

And the chief problem still remains, which has not been addressed until this 
patch (as far as I know):
In Python, it is impossible to combine existing context managers into a nested 
one.  ExitStack may address a use case of nested context managers, but it 
doesn't address the basic problem.

ContextManagerExit comes with its own nice little features, too.  Now you can 
write:

@contextlib.contextmanager:
def if_ctxt(condition):
if condition:
yield

#hey look! an if statement as a with statement!
with if_ctxt(condition):
do_work

This can easily be extended, where a context manager can both manage context, 
_and_ provide optional execution of its block.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[R. David Murray]

R. David Murray added the comment:

Raising it on python-ideas sounds like a good idea, then.

I must admit that I don't understand what you mean by combining existing 
context managers into a nested one that isn't addressed by ExitStack.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Simply put, there is no way in the language to nest two context managers, even 
though we have full access to their implementation model, i.e. can call 
__enter__ and __exit__ manually.  This reflects badly (pun intended) on 
Python's reflection and introspection capabilities.

If context managers are to be first class entities in the language, then you 
ought to be able to write absract code using them, and
assemble complex ones out of simple ones.  Hypothetical code here:

def nest(a, b):
# currently not possible
return c

def run_with_context(ctxt, callable):
# abstract executor
with ctxt:
return callable()

run_with_context(nested(a,b), callable)

ExitStack address one use case that contextlib.nested was supposed to solve, 
namely the cleanup of a dynamic sequence of context managers.  But it does this 
no by creating a new manager, but by providing a programming pattern to follow. 
 In that sensse, the multiple context manager syntax (with (a, b, c): ) is also 
a hack because it provides language magic to perform what you ought to be able 
to do dynamically...

Does this makes sense?

Anyway, by providing the ContextManagerExit exception, then sufficient 
flexibility is added to the context manager mechanism that at least the use 
case of nested() becomes possible.

Context managers are really interesting things.  I was inspired by Raymond 
Hettinger's talk last pycon to explore their capabilities and this is one of 
the things I came up with :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Nick Coghlan]

Nick Coghlan added the comment:

I pitched the idea of making it possible to skip the with statement body
quite some time ago, and Guido convinced me it was a bad idea for much the
same reason he chose PEP 343 over his original PEP 340 design: allowing
suppression of exceptions from __enter__ hides local control flow by
blurring the boundaries between with and if statements.

Regarding nested, we killed that because it was a bug magnet for context
managers that acquire the resource in __init__ (like file objects), not
because it didn't work.
It's trivial to recreate that API on top of ExitStack if you like it,
though. The only thing that doesn't work (relative to actual nested with
statements) is suppressing exceptions raised inside __enter__ methods.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Hi there.
allowing
suppression of exceptions from __enter__ hides local control flow by
blurring the boundaries between with and if statements.

I'm not sure what this means.  To me, it is a serious language design flaw that 
you can write a context manager, and it has a well known interface that you can 
invoke manually, but that you cannot take two existing context managers and 
assemble them into a nested one, correctly, however much you wiggle.
In my mind, allowing context managers to skip the managed body breaks new 
ground.  Both, by allowing this combination to be possible.  And also by 
opening up new and exciting applications for context managers.  If you saw 
Raymond's talk last Pycon, you should feel inspired to do new and exciting 
things with them.

the bug-magnet you speak of I already addressed in my patch with 
nested-delayed, more as a measure of completeness (address both the problems 
that old nested had.  The more serious bug (IMHO) is the suppression of 
__enter__ exceptions.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18677] Enhanced context managers with ContextManagerExit and None

[Eric Snow]

Eric Snow added the comment:

Nick was probably talking about what is further elaborated in PEP 343.  I'd 
recommend taking a particular look at the Motivation and Summary section 
regarding flow control macros.

--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18677
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com