Re: [Python-Dev] Possible context managers in stdlib
Skip Montanaro wrote: After seeing so many messages about with statements my eyes began to glaze over, so I stopped following that thread. Then I saw mention of context manager with no reference to any PEPs or to the with statement to provide context. The main outcome of the PEP 343 terminology discussion was some proposed documentation I put on the Sourceforge patch tracker ([1]). The patch is currently assigned to Raymond (since he started the terminology discussion) but any other reviews would be welcome. Since SF currently doesn't want to play, and the proposed documentation isn't that long, I've included the latest version below for anyone who wants to read it. None of the context-providing messages seemed to have been indexed by Google when I checked, so searching for Python context manager failed to return anything useful. Hence the post. Google appears to have spidered the list archives some time today, so anyone else doing the same search should get some relevant hits. Cheers, Nick. [1] http://www.python.org/sf/1234057 == With Statements and Context Management A frequent need in programming is to ensure a particular action is taken after a specific section of code has been executed (such as closing a file or releasing a lock). Traditionally, this is handled using 'try'/'finally' statements. However, that approach can lead to the reproduction of non-trivial amounts of boilerplate whenever the action needs to be invoked. A simpler way to achieve this in Python is to use the 'with' statement along with the appropriate context manager. Context managers define an action which is taken to enter the context and a second action to exit the context (usually restoring the environment that existed before the context was entered). The 'with' statement ensures that the context is entered and exited at the appropriate times (that is, before and after the execution of the suite contained in the 'with' statement). The precise behaviour of the 'with' statement is governed by the supplied context manager - an object which supports the context management protocol. This protocol consists of two methods: __enter__(self): Context managers use this method to enter the desired context before the execution of the contained suite. This method is called without arguments before execution of the contained suite starts. If the 'as' clause of the 'with' statement is used, the value returned from this method is assigned to the specified target. Many context managers will return self from this method, but returning a different object may make sense for some managers (for example, see the 'closing' suite manager described below). __exit__(self, exc_type, exc_value, exc_traceback): Context managers use this method to exit the context after execution of the contained suite. This method is called after execution of the contained suite is completed. If execution completed due to an exception, the details of that exception are passed as arguments. Otherwise, all three arguments are set to None. If exception details are passed in, and this method returns without incident, then the original exception continues to propagate. Otherwise, the exception raised by this method will replace the original exception. Using Contexts to Manage Resources The simplest use of context management is to strictly control the handling of key resources (such as files, generators, database connections, synchronisation locks). These resource managers will generally acquire the resource in their __enter__ method, although some resource managers may accept the resource to be managed as an argument to the constructor or acquire it during construction. Resource managers will then release the resource in their __exit__ method. For example, the following context manager allows prompt closure of any resource with a 'close' method (e.g. a generator or file): class closing(object): def __init__(self, resource): self.resource = resource def __enter__(self): return self.resource def __exit__(self, *exc_info): self.resource.close() with closing(my_generator()) as g: # my_generator() is assigned to g via call to __enter__() for item in g: print item # g is closed as the with statement ends Some resources (such as threading.Lock) support the context management protocol natively, allowing them to be used directly in 'with' statements. The meaning of the established context will depend on the specific resource. In the case of threading.Lock, the lock is acquired by the __enter__ method, and released by the __exit__ method. with the_lock: # Suite is executed with the_lock held # the_lock is released as the with statement ends More
Re: [Python-Dev] Possible context managers in stdlib
FWIW, I've updated PEP 343 to use @contextmanager and class ContextWrapper. Please proofread. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Nick Coghlan [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] The main outcome of the PEP 343 terminology discussion was some proposed documentation I put on the Sourceforge patch tracker ([1]). Is this a proposal for the Language Reference manual? [1] http://www.python.org/sf/1234057 == With Statements and Context Management A frequent need in programming ... A simpler way to achieve this in Python is to use the 'with' statement along with the appropriate context manager. Somewhere about here we need the syntax itself. Context managers define an... the contained suite starts. If the 'as' clause of the 'with' Else this does not mean much. ... The simplest use of context management is to strictly control the handling of key resources (such as files, generators, database connections, synchronisation locks). I have a little trouble seeing generators (as opposed to iterables) as resources needing management. For example, the following context manager allows prompt closure of any resource with a 'close' method (e.g. a generator or file): And I was not aware that they had close methods. You mean a iterable (not just a file) with both an associated generator and a close? Or are generators gaining close methods (which make no sense to me). Or are you using 'generator' in a different sense? class closing(object): def __init__(self, resource): self.resource = resource def __enter__(self): return self.resource def __exit__(self, *exc_info): self.resource.close() with closing(my_generator()) as g: # my_generator() is assigned to g via call to __enter__() for item in g: print item # g is closed as the with statement ends To me, this should be with closing(my_iterable())... with 'for' calling g.__iter__ to get the iterator that is possibly a generator. Otherwise, I don't understand it. The rest is pretty clear. Terry J. Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Terry Reedy wrote: Nick Coghlan [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] The main outcome of the PEP 343 terminology discussion was some proposed documentation I put on the Sourceforge patch tracker ([1]). Is this a proposal for the Language Reference manual? No - it's for an entry in the Library reference under 'built-in types', as a sibling to the current documentation of the iteration protocol. The 'with' statement itself would have to be documented along with the rest of the grammar. A simpler way to achieve this in Python is to use the 'with' statement along with the appropriate context manager. Somewhere about here we need the syntax itself. I'm not sure. We don't reproduce the 'for' loop syntax in the documentation of iterators, so should we reproduce the 'with' statement syntax in the documentation of context managers? Again, modelling on the existing documentation of the iteration protocol, I would expect the statement syntax to be introduced in the tutorial (e.g. as part of Section 8.6, where try/finally is introduced). Context managers define an... the contained suite starts. If the 'as' clause of the 'with' Else this does not mean much. ... The simplest use of context management is to strictly control the handling of key resources (such as files, generators, database connections, synchronisation locks). I have a little trouble seeing generators (as opposed to iterables) as resources needing management. PEP 342 adds this, in order to allow 'yield' inside tyr/finally blocks. For example, the following context manager allows prompt closure of any resource with a 'close' method (e.g. a generator or file): And I was not aware that they had close methods. You mean a iterable (not just a file) with both an associated generator and a close? Or are generators gaining close methods (which make no sense to me). Or are you using 'generator' in a different sense? Sorry - these docs assume PEP 342 has been implemented, so generator's have close() methods. I was trying to steer clear of files, since we don't know yet whether there is going to be an opening or closing implementation in the standard library, or whether files will become context managers. The latter is my preference, but Guido didn't seem too keen on the idea last time it was brought up. class closing(object): def __init__(self, resource): self.resource = resource def __enter__(self): return self.resource def __exit__(self, *exc_info): self.resource.close() with closing(my_generator()) as g: # my_generator() is assigned to g via call to __enter__() for item in g: print item # g is closed as the with statement ends To me, this should be with closing(my_iterable())... with 'for' calling g.__iter__ to get the iterator that is possibly a generator. Otherwise, I don't understand it. The rest is pretty clear. Terry J. Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40email.com -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Skip Montanaro [EMAIL PROTECTED] writes: Ummm... What's a context manager? Something that goes with ... as var: ^ here If you have a better name, feel free to suggest it, but please catch up on python-dev first (it's been discussed to unconsciousness, if not quite death, in the last week or so). Cheers, mwh -- dash i am trying to get Asterisk to work dash it is stabbing me in the face dreid yes ... i seem to recall that feature in the documentation -- from Twisted.Quotes ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Ummm... What's a context manager? Michael Something that goes Michael with ... as var: Michael ^ here Michael If you have a better name, feel free to suggest it, but please Michael catch up on python-dev first (it's been discussed to Michael unconsciousness, if not quite death, in the last week or so). After seeing so many messages about with statements my eyes began to glaze over, so I stopped following that thread. Then I saw mention of context manager with no reference to any PEPs or to the with statement to provide context. None of the context-providing messages seemed to have been indexed by Google when I checked, so searching for Python context manager failed to return anything useful. Hence the post. BTW, context manager seems fine to me in that context... Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
On 7/8/05, James Y Knight [EMAIL PROTECTED] wrote: It is a really bad idea to codify the practice of modifying non- threadlocal global state like sys.std[in|out|err] and current directory with a context manager. A user can do it to themselves now, true, but by putting a context manager for it in the stdlib, you make it look like it'd be a local modification when it is not. I can only see confusion resulting from this. Users will inevitably try to use it like with sys.redirected_stderr(f): Which is of course a bug -- print doesn't go to stderr. Not sure if you meant this as an illustration of a typical bug or whether you meant sys.redirect_stdout(f). print hello print there instead of explicitly writing to f with print or write(). And that is just a terribly bad idea. It looks pretty, yes, but unless stdinouterr are made thread-local, it's just asking for trouble. Boy, do you have Java (or multi-threading) on your mind. A lot of Python programs are single-threaded, and this idiom has been promoted by examples for many years; I see nothing wrong with it in single-threaded code. I sure hope the world doesn't evolve to one where most programs have to be multi-threaded to be useful! Multi-threading is such a nightmare to program that we should really look for paradigms that hide it completely rather than trying to raise programmers who will write correct multi-threaded programs naturally; the latter just ain't gonna happen. A major use case for this, BTW, is to take *existing* Python code that was written using print statements and wrap it in something that captures its output. The f.write() or printf solutions won't work there... -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
I wrote: I agree with Barry. Not only should they be in the stdlib, but they should have very clear warnings in their docstrings and other documentation that state that they are ONLY safe to use in single-threaded programs. This achieves two things: it makes them available to those who need them (not everyone uses threads!), and it rather forcefully makes the point that it's NOT usually a good idea to modify global state info in a context manager because doing so is not generally threadsafe. Nick Coghlan replies: Wouldn't they be able to actually emit a warning at run-time if they're used in a multi-threaded program? That would be even better motivation for including them, IMO. I don't think that would be desirable. These things CAN be useful in a multi-threaded program if you know what you're doing. One common example would be to use them only from the main thread. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
On 7/8/05, Reinhold Birkenfeld [EMAIL PROTECTED] wrote: Hi, I compiled a list of some possible new context managers that could be added to the stdlib. Introducing a new feature should IMO also show usage of it in the distribution itself. That wasn't done with decorators (a decorators module is compiled at the moment, if I'm right), but with context managers, there's certainly room to add some. Of course, my list is excessive, it's only some ideas I got by flying over the stdlib docs. I think that is a good point about decorators. While none of us obviously had extensive experience with decorators when they went into the language we at least knew of a couple use cases that would have been handy and it probably would have been good to have examples for people to work off of. I bet iterators have gotten more play thanks to itertools and all of the built-ins that work with them. I think having basic context managers in a stdlib module that we know for a fact that will be handy is a good idea. We should keep the list short and poignant, but we should have something for people to work off of. The ones I like below for a 'context' module are: * builtins: with open/file * sys: with sys.redirected_std[in|out|err] * decimal: with decimal.Context * os: with os.current_directory * mutex: with mutexobj * threading: with threading.Lock with threading.Condition with threading.Event * bz2/zipfile/tarfile: with ...open -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
On Jul 8, 2005, at 4:46 PM, Brett Cannon wrote: I think having basic context managers in a stdlib module that we know for a fact that will be handy is a good idea. We should keep the list short and poignant, but we should have something for people to work off of. The ones I like below for a 'context' module are: * builtins: with open/file * sys: with sys.redirected_std[in|out|err] * decimal: with decimal.Context * os: with os.current_directory * mutex: with mutexobj * threading: with threading.Lock with threading.Condition with threading.Event * bz2/zipfile/tarfile: with ...open It is a really bad idea to codify the practice of modifying non- threadlocal global state like sys.std[in|out|err] and current directory with a context manager. A user can do it to themselves now, true, but by putting a context manager for it in the stdlib, you make it look like it'd be a local modification when it is not. I can only see confusion resulting from this. Users will inevitably try to use it like with sys.redirected_stderr(f): print hello print there instead of explicitly writing to f with print or write(). And that is just a terribly bad idea. It looks pretty, yes, but unless stdinouterr are made thread-local, it's just asking for trouble. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
On Fri, 2005-07-08 at 16:24, Reinhold Birkenfeld wrote: I compiled a list of some possible new context managers that could be added to the stdlib. I agree with Brett and Phillip that a few well-chosen context managers would make sense in the stdlib both for convenience and for example purposes. Keep the list short and sweet. Also, I'd like to see some higher-level cm's in the mix, perhaps implementing concepts like Java's synchronized (perhaps even being named 'with synchronization'). Thinking about the types of code I write over and over again, I think I disagree (slightly) with James about the global state thing. While I agree that 'with redirected_stdio' isn't too useful in the face of print, I very often have to write try/finally protections around temporary settings of the cwd, the umask, and other global process values. I'd love to see cm's for those constructs in the stdlib. -Barry signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Phillip J. Eby wrote: At 10:24 PM 7/8/2005 +0200, Reinhold Birkenfeld wrote: with sys.trace Note that it's currently not possible to inspect the trace/profile hooks from Python code, only from C, so that might be, um, interesting to implement. That was beyond my short view... if it can't be implemented, it won't. * pprint: with pprint.printer (used for print) Interesting; I'm not sure if I like it. * os: with os.current_directory What does this do? Oh, I get it. The name's not very obvious. I would've expected a more imperative name, something like 'with os.setcwd()' or 'with os.pushdir()'. I didn't think about the names too long. ;) with os.modified_env with os.umask/uid/gid etc. Yeah, again I would expect more verbish names, but these are at least easier to grasp than current_directory was. Names can be found, of course. * curses: with curses.wrapper with logging.Logger * tty: with tty.raw with tty.cbreak * cgitb: with cgitb.enabled What do these do? curses: curses.wrapper currently is a function which restores sane terminal settings after the curses program bails out. It could be a context manager too. Similar is tty.raw and tty.cbreak. These switch terminal modes, and in case of an uncaught exception the terminal will stay in this state. Context managers would restore the sane mode on exit. logging.Logger: hm, I didn't think about that properly. cgitb: enables or disables the full traceback view (Ok, it wouldn't be so useful). Reinhold -- Mail address is perfectly valid! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
I compiled a list of some possible new context managers that could be added to the stdlib. Introducing a new feature should IMO also show usage of it in the distribution itself. That wasn't done with decorators (a decorators module is compiled at the moment, if I'm right), but with context managers, there's certainly room to add some. Of course, my list is excessive, it's only some ideas I got by flying over the stdlib docs. The PEP contains plenty of examples. If you're really feeling the need, add something to the demo directory. For the most part, the applications need to work themselves out over time through wikis, individual patch submissions, ASPN recipes, third-party apps, etc. The natural evolution of best practices tends to get thwarted by prematurely using the standard library to cast a particular solution in stone. This doesn't preclude individual patches such as a context manager for decimal.Context objects. Each proposal should be considered on its own merits rather than as a part of an overall effort to ram a bunch of these into the standard library.Over time, plenty of these will sprout-up. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
James Y Knight writes: It is a really bad idea to codify the practice of modifying non- threadlocal global state like sys.std[in|out|err] and current directory with a context manager. Barry Warsaw responds: Thinking about the types of code I write over and over again, I think I disagree (slightly) with James about the global state thing. While I agree that 'with redirected_stdio' isn't too useful in the face of print, I very often have to write try/finally protections around temporary settings of the cwd, the umask, and other global process values. I'd love to see cm's for those constructs in the stdlib. I agree with Barry. Not only should they be in the stdlib, but they should have very clear warnings in their docstrings and other documentation that state that they are ONLY safe to use in single-threaded programs. This achieves two things: it makes them available to those who need them (not everyone uses threads!), and it rather forcefully makes the point that it's NOT usually a good idea to modify global state info in a context manager because doing so is not generally threadsafe. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Raymond Hettinger wrote: I compiled a list of some possible new context managers that could be added to the stdlib. Introducing a new feature should IMO also show usage of it in the distribution itself. That wasn't done with decorators (a decorators module is compiled at the moment, if I'm right), but with context managers, there's certainly room to add some. Of course, my list is excessive, it's only some ideas I got by flying over the stdlib docs. The PEP contains plenty of examples. If you're really feeling the need, add something to the demo directory. For the most part, the applications need to work themselves out over time through wikis, individual patch submissions, ASPN recipes, third-party apps, etc. The natural evolution of best practices tends to get thwarted by prematurely using the standard library to cast a particular solution in stone. This doesn't preclude individual patches such as a context manager for decimal.Context objects. Each proposal should be considered on its own merits rather than as a part of an overall effort to ram a bunch of these into the standard library.Over time, plenty of these will sprout-up. As with decorators? Sure, plenty have sprouted up, but delivering them one release after the feature itself (if the decorators module makes it into 2.5) seems a little bit like lagging behind -- especially when good and useful examples exist. Of course, not all of these in my list are good ideas to implement. Reinhold -- Mail address is perfectly valid! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Ummm... What's a context manager? Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible context managers in stdlib
Michael Chermside wrote: I agree with Barry. Not only should they be in the stdlib, but they should have very clear warnings in their docstrings and other documentation that state that they are ONLY safe to use in single-threaded programs. This achieves two things: it makes them available to those who need them (not everyone uses threads!), and it rather forcefully makes the point that it's NOT usually a good idea to modify global state info in a context manager because doing so is not generally threadsafe. Wouldn't they be able to actually emit a warning at run-time if they're used in a multi-threaded program? That would be even better motivation for including them, IMO. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
