Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
Michael Chermside wrote: > If I write > > with foo: >BLOCK > > where I should have written > > with locked(foo): >BLOCK > > ...it silently "succeeds" by doing nothing. I CLEARLY intended to > do the appropriate cleanup (or locking, or whatever), but it doesn't > happen. Ah, thanks. Like Guido, I had something in the back of my head saying it didn't like the idea, but I couldn't figure out the reason. I think you just nailed it. Plus, there is a nice alternative which is to provide a 'clean it up if it needs it' resource in the standard library: class resource(object): def __init__(self, obj): self.obj = obj self.enter = getattr(obj, "__enter__", None) self.exit = getattr(obj, "__exit__", None) def __enter__(self): if self.enter is not None: self.enter() # For consistency, always return the object return self.obj def __exit__(self, *exc_info): if self.exit is not None: self.exit(*exc_info) Then 'I don't know if this needs cleaning up or not' can be written: with resource(foo): # If foo needs cleaning up, it will be. A refinement would provide the option to specify the enter/exit methods directly: class resource(object): def __init__(self, obj, *other_args): self.obj = obj if other_args: if len(other_args) != 2: raise TypeError("need 1 or 3 arguments") self.enter = args[0] self.exit = None self.exit_no_args = args[1] else: self.enter = getattr(obj, "__enter__", None) self.exit = getattr(obj, "__exit__", None) self.exit_no_args = None def __enter__(self): if self.enter is not None: self.enter() # For consistency, always return the object return self.obj def __exit__(self, *exc_info): if self.exit is not None: self.exit(*exc_info) elif self.exit_no_args is not None: self.exit() That would let any object with a standard 'clean me up method' be easily used in a with statement: with resource(bar, None, bar.clear): # bar will be cleared when we're done Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
Guido writes: [a rather silly objection to Phillip's proposal that 'with x:' is a no-op when x lacks __enter__ and __exit__] > I know this is not a very strong argument, but my gut tells me this > generalization of the with-statement is wrong, so I'll stick to it > regardless of the strength of the argument. The real reason will come > to me. Perhaps the real reason is that it allows errors to pass silently. If I write with foo: BLOCK where I should have written with locked(foo): BLOCK ...it silently "succeeds" by doing nothing. I CLEARLY intended to do the appropriate cleanup (or locking, or whatever), but it doesn't happen. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 -Abstract Block Redux)
[I apologize in advance if this sounds a bit disjointed... I started to argue one thing, but by the end had convinced myself of the opposite, and I re-wrote the email to match my final conclusion.] Guido writes: > About deleting VAR I have mixed feelings. [...] > I think that, given that we let the for-loop variable survive, we > should treat the with-statement variable the same way. We said the same thing about the variable in list comprehensions and it's now obvious that it should NEVER have been allowed to escape it's scope. But the key difference is that list comprehensions are EXPRESSIONS, and for and 'with' are STATEMENTS. Expressions shouldn't modify the local environment, statements often do. Of course, that argument _permits_ not deleting VAR, but doesn't recomend in favor of it. My first thought for ideal behavior was that if VAR was previously defined (eg: a global, an existing attribute of some object, etc), then it should not be 'del''ed afterward. But VAR was newly created by the 'with' statement then we WOULD del it to keep the namespace "neat". Trouble is, that's FAR too complex, and depends on a distinction Python has not used before (although it's nearly the same as the property that controls the meaning of globally declared variables). My next thought was to just allow 'with' statements to introduce their own "scope"... the meaning that the VAR variable takes on within a 'with' statement is not propogated outside the scope of the statement. But imagine trying to implement this in CPython... don't forget details like supporting locals(). If it's too hard to do, then it's probably not the right solution. So then I thought "Well, what's the harm in letting the variable survive the 'with' statement?" I'm a big fan of keeping namespaces "clean", but it's just not important enough to incurr other penalties. So in this case, I (reluctantly, after giving myself quite a talking-to) favor having the 'with' statement with VAR create said variable in the appropriate scope as a side-effect, much like 'for'. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
At 12:48 PM 5/18/2005 +1200, Greg Ewing wrote: >Nick Coghlan wrote: > > >the_stmt = EXPR1 > >stmt_enter = getattr(the_stmt, "__enter__", None) > >stmt_exit = getattr(the_stmt, "__exit__", None) > > > >if stmt_enter is None: > >VAR1 = the_stmt > >else: > >VAR1 = stmt_enter() > >If we're doing this, it might be better if VAR were simply >bound to EXPR in all cases. Otherwise the ability to liberally >sprinkle with-statements around will be hampered by uncertainty >about what kind of object VAR will end up being. It'll be whatever kind of object you should get when you're using EXPR as a resource. :) In other words, it looks like normal duck typing to me: what you get when you access something.quack() is whatever 'something' thinks you should get. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
And we're back at PEP 310 and you can't really write opening() as a generator. On 5/17/05, Greg Ewing <[EMAIL PROTECTED]> wrote: > Nick Coghlan wrote: > > >the_stmt = EXPR1 > >stmt_enter = getattr(the_stmt, "__enter__", None) > >stmt_exit = getattr(the_stmt, "__exit__", None) > > > >if stmt_enter is None: > >VAR1 = the_stmt > >else: > >VAR1 = stmt_enter() > > If we're doing this, it might be better if VAR were simply > bound to EXPR in all cases. Otherwise the ability to liberally > sprinkle with-statements around will be hampered by uncertainty > about what kind of object VAR will end up being. > > Greg > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
Nick Coghlan wrote: >the_stmt = EXPR1 >stmt_enter = getattr(the_stmt, "__enter__", None) >stmt_exit = getattr(the_stmt, "__exit__", None) > >if stmt_enter is None: >VAR1 = the_stmt >else: >VAR1 = stmt_enter() If we're doing this, it might be better if VAR were simply bound to EXPR in all cases. Otherwise the ability to liberally sprinkle with-statements around will be hampered by uncertainty about what kind of object VAR will end up being. Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
At 07:17 AM 5/17/2005 -0700, Guido van Rossum wrote: >The compiler must generate both code paths but one is wasted. Not if the default behavior is encapsulated in PyResource_Enter() (returning the object if no __enter__) and PyResource_Exit() (a no-op if no __exit__). You're going to have to have the branch and test for slot presence *somewhere*. >I know this is not a very strong argument, It's not even an actual argument. :) >but my gut tells me this >generalization of the with-statement is wrong, so I'll stick to it >regardless of the strength of the argument. The real reason will come >to me. I don't know about anybody else, but I'd prefer to hear that your gut tells you it's wrong, because that at least tells me there's no point in arguing. Pseudo-reasons just make me think there's something to discuss. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
At 11:03 PM 5/17/2005 +1000, Nick Coghlan wrote: >Do you mean translating this: > >with EXPR1 as VAR1: >BLOCK1 > >To something along the lines of: > >the_stmt = EXPR1 >stmt_enter = getattr(the_stmt, "__enter__", None) >stmt_exit = getattr(the_stmt, "__exit__", None) > >if stmt_enter is None: >VAR1 = the_stmt >else: >VAR1 = stmt_enter() > >if stmt_exit is None: >BLOCK1 >else: >exc = (None, None, None) >try: >try: >BLOCK1 >except: >exc = sys.exc_info() >raise >finally: >stmt_exit(*exc) Essentially, yes; although I was actually suggesting that there be PyResource_Enter() and PyResource_Exit() C APIs that would supply the default behaviors if the tp_resource_enter and tp_resource_exit slots were missing from the object's type. But that's an implementation detail. >It has a certain elegance - you can switch from using an object that needs >finalisation to one that doesn't without having to change your code. > >And library code can safely ensure finalisation without having to check >whether >the object needs it or not - that check becomes an inherent part of the with >statement. Precisely. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
[Nick Coghlan (replying to Phillip)] > Do you mean translating this: > >with EXPR1 as VAR1: >BLOCK1 > > To something along the lines of: > >the_stmt = EXPR1 >stmt_enter = getattr(the_stmt, "__enter__", None) >stmt_exit = getattr(the_stmt, "__exit__", None) > >if stmt_enter is None: >VAR1 = the_stmt >else: >VAR1 = stmt_enter() > >if stmt_exit is None: >BLOCK1 >else: >exc = (None, None, None) >try: >try: >BLOCK1 >except: >exc = sys.exc_info() >raise >finally: >stmt_exit(*exc) -1. The compiler must generate both code paths but one is wasted. I know this is not a very strong argument, but my gut tells me this generalization of the with-statement is wrong, so I'll stick to it regardless of the strength of the argument. The real reason will come to me. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
Phillip J. Eby wrote: > I'm suggesting that we simply take Nick's proposal to its logical > conclusion, and allow any object to be usable under "with", since it > does not create any problems to do so. (At least, none that I can > see.) A redundant 'with' does no harm; in the worst case it's just a > hint to the reader about the scope within which an expression is used > within the current fuction body. Do you mean translating this: with EXPR1 as VAR1: BLOCK1 To something along the lines of: the_stmt = EXPR1 stmt_enter = getattr(the_stmt, "__enter__", None) stmt_exit = getattr(the_stmt, "__exit__", None) if stmt_enter is None: VAR1 = the_stmt else: VAR1 = stmt_enter() if stmt_exit is None: BLOCK1 else: exc = (None, None, None) try: try: BLOCK1 except: exc = sys.exc_info() raise finally: stmt_exit(*exc) It has a certain elegance - you can switch from using an object that needs finalisation to one that doesn't without having to change your code. And library code can safely ensure finalisation without having to check whether the object needs it or not - that check becomes an inherent part of the with statement. I'm ambivalent about this one - I see some benefit to it, but there's something niggling at the back of my brain that doesn't like it (nothing I can point to in particular, though). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
At 11:20 AM 5/16/2005 -0700, Guido van Rossum wrote: >I think the issue here is not implementation details but whether it >follows a certain protocol. IMO it's totally acceptable to require >that the expression used in a with-statement support an appropriate >protocol, just like we require the expression used in a for-loop to be >an iterable. Perhaps I didn't explain well. I mean that whether early release of a resource is desirable, depends on the resource. For example, consider a file versus a StringIO. If files support resource release and StringIO's don't, this pollutes client code with implementation knowledge. Therefore, the penalty for people trying to clean up resources early is that they either pollute their client code with checking to see if things are 'with'-able (which seems insane), or else adding empty __enter__ and __exit__ methods to things so that their consumers can just use "with" whenever they want to scope a resource, whether the resource actually needs it or not. I'm suggesting that we simply take Nick's proposal to its logical conclusion, and allow any object to be usable under "with", since it does not create any problems to do so. (At least, none that I can see.) A redundant 'with' does no harm; in the worst case it's just a hint to the reader about the scope within which an expression is used within the current fuction body. > > This insight may actually be true regardless of what generators do or don't > > do; the point is that if you change from using a generator to a built-in > > iterator type, you shouldn't have to change every place you were using the > > 'with' blocks to work again. > >Huh? The with-statement doesn't loop, and its use of generators is >such that I don't see how you could ever replace it with a built-in >iterator. I'm referring here to Nick's proposal of doing things like this: with some_function() as items: for item in items: # etc. To ensure that a *normal* generator is finalized. This isn't about generator templates. So, if you were ever to change 'some_function()' to return a list instead of being a generator, for example, this code would no longer work. >About deleting VAR I have mixed feelings. I appreciate the observation >that it's most likely dead after the with-statement is over, but I'm >not sure that explicit deletion is correct. Remember that VAR can be >an arbitrary assignment target, which means it could be a global, or a >tuple, or an indexed list or dict item, or some object's attribute (I >hesitate to write "etc." after such an exhaustive list :-). Argh. I forgot about that. :( On the other hand, I haven't seen any examples where arbitrary assignment targets were actually used, so perhaps it could be limited to local variables (and possibly-nested tuples thereof). But I also partly agree with the false-sense-of-security thing, too, so I'm not sure on this either now that you point out the issue. It also makes me wonder about something like this: with open("foo") as f: def callback(): return f.read() which can't behave sanely if 'callback' is used outside the 'with' block. Of course, it won't work right even if 'f' remains bound to the file object, but this does seem to open new implementation complexities to get all these pieces to work right. :( > > should also be short for this (in the case where some_expr() has no > > __enter__ or __exit__ methods): > > > > foo = some_expr() > > try: > > # etc. > > finally: > > del foo > >-1 on this. You're trying to pack too much into a single statement. Note that I was just leaving out the rest of the standard PEP 3XX expansion above, just highlighting the proposed 'del' portion of the expansion. That is, I wasn't proposing an alternate expansion, just showing the effect of the other parts of the expansion being skipped because of null __enter__/__exit__. > > And that could be a useful thing for many existing object types, without > > even updating them for PEP 34[0-9]. :) It wouldn't be *as* useful for > > non-CPython implementations, but presumably by the time those > > implementations catch up, more code will be out there with > > __enter__/__exit__ methods. > >Most likely it would just give an implementation-dependent false sense >of security. I think I prefer to see the glass half-full here; that is, I think it increases the safety during a transitional period. But the scoping of arbitrary VAR expansions makes things trickier, so I'll think about this some more. >-1 on the whole thing (and noting that this would be an easy extension >if at some point in the future we change our mind). Your post has made me think that I might have the concept backwards. Instead of guaranteeing that the variable goes out of scope at the end of the block, what a 'with' block actually guarantees (implicit in all of the PEP 34x variants) is that EXPR will
Re: [Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
Nick's PEP 3XX is here: http://members.iinet.net.au/~ncoghlan/public/pep-3XX.html [Phillip J. Eby] > Anyway, I took a look at it, and I mostly like it. I like the beginning of Nick's PEP too, since its spec for the with statement is now identical to PEP 343 apart from the keyword choice (and I'm leaving that open for a public vote). The fundamental difference now is that Nick proposes an exception injection and finalization API for generators. I think that ought to be done as a separate PEP, since it's relatively separate from the do-or-with-statement (even though it's clear that PEP 343 would be the first to benefit). Maybe Nick can strip his PEP (or start a new one) to only addresses generator exception injection and generator finalization. I think he's on to something but it needs to be spec'ed more exactly. Continuing with Phillip...: > There appears to be an > error in "Deterministic generator finalisation" (maybe you already know > this): the _inject_exception() should be called with exc_info, not > TerminateIteration, and it should swallow StopIteration instead of > TerminateIteration. IOW, I think it should look like this: > > def __exit__(self, *exc_info): > try: > self._inject_exception(*exc_info) > except StopIteration: > pass > > Hm. Oh wait, I just realized - you don't mean this at all. You're > describing a use of generators as non-templates. Ugh. I think that might > lead to confusion about the semantics of 'with' and generators. I'll have > to think about it some more, but my second impression after a little bit of > thought is that if you're going to do this, then you should be allowed to > use 'with' with any object, using the object as VAR if there's no > __enter__. My reasoning here is that it then makes it possible for you to > use arbitrary objects for 'with' without needing to know their > implementation details. It should be harmless to use 'with' on objects > that don't need it. I think the issue here is not implementation details but whether it follows a certain protocol. IMO it's totally acceptable to require that the expression used in a with-statement support an appropriate protocol, just like we require the expression used in a for-loop to be an iterable. > This insight may actually be true regardless of what generators do or don't > do; the point is that if you change from using a generator to a built-in > iterator type, you shouldn't have to change every place you were using the > 'with' blocks to work again. Huh? The with-statement doesn't loop, and its use of generators is such that I don't see how you could ever replace it with a built-in iterator. > A further part of this insight: perhaps the 'with' block translation should > include a 'del VAR' in its finally block, not to mention the equivalent of > 'del stmt_enter,stmt_exit'. In other words, the binding of VAR should not > escape the 'with' block. I've seen Nick's stmt_enter and stmt_exit as implementation details that probably would be done differently by a real implementation; personally I don't mind getting the TypeError about a missing __exit__ only when __exit__ is called (after all if this happens the program is too buggy to care much about the precise semantics) so this is what PEP 343 does. About deleting VAR I have mixed feelings. I appreciate the observation that it's most likely dead after the with-statement is over, but I'm not sure that explicit deletion is correct. Remember that VAR can be an arbitrary assignment target, which means it could be a global, or a tuple, or an indexed list or dict item, or some object's attribute (I hesitate to write "etc." after such an exhaustive list :-). Example 8 in PEP 340 shows a use case where at least one of the variables (the error value) could meaningfully survive the block. I think that, given that we let the for-loop variable survive, we should treat the with-statement variable the same way. A good compiler can see that it's dead if it's unused further and delete it earlier, but I don't think we need more -- especially since in a GC'ed implementation like Jython or IronPython, deleting it doesn't do us much good. > This would mean that for existing types that use > __del__ for cleanup (e.g. files and sockets), then 'with open("file") as f' > would automatically ensure closing under CPython (but other implementations > would be allowed to wait for GC). In other words, I'm saying that this: > > with some_expr() as foo: > # etc. > > should also be short for this (in the case where some_expr() has no > __enter__ or __exit__ methods): > > foo = some_expr() > try: > # etc. > finally: > del foo -1 on this. You're trying to pack too much into a single statement. > And that could be a useful thing for many existing object types, without > even updating them for PEP 34[0-9]. :) It wouldn't be *as* useful for > non-CPython implementations, but
[Python-Dev] Simpler finalization semantics (was Re: PEP 343 - Abstract Block Redux)
At 06:56 PM 5/16/2005 +1000, Nick Coghlan wrote: >Anyway, I think it's stable enough now that I can submit it to be put up on >www.python.org (I'll notify the PEP editors directly once I fix a couple of >errors in the current version - like the missing 'raise' in the statement >semantics. . .). If you have developer checkin privileges, it's best to get a PEP number sooner rather than later, if the PEP shows any signs of viability at all. Once you're in the PEP infrastructure people can subscribe to get notified when you change it, read the revision history, and so on. Anyway, I took a look at it, and I mostly like it. There appears to be an error in "Deterministic generator finalisation" (maybe you already know this): the _inject_exception() should be called with exc_info, not TerminateIteration, and it should swallow StopIteration instead of TerminateIteration. IOW, I think it should look like this: def __exit__(self, *exc_info): try: self._inject_exception(*exc_info) except StopIteration: pass Hm. Oh wait, I just realized - you don't mean this at all. You're describing a use of generators as non-templates. Ugh. I think that might lead to confusion about the semantics of 'with' and generators. I'll have to think about it some more, but my second impression after a little bit of thought is that if you're going to do this, then you should be allowed to use 'with' with any object, using the object as VAR if there's no __enter__. My reasoning here is that it then makes it possible for you to use arbitrary objects for 'with' without needing to know their implementation details. It should be harmless to use 'with' on objects that don't need it. This insight may actually be true regardless of what generators do or don't do; the point is that if you change from using a generator to a built-in iterator type, you shouldn't have to change every place you were using the 'with' blocks to work again. A further part of this insight: perhaps the 'with' block translation should include a 'del VAR' in its finally block, not to mention the equivalent of 'del stmt_enter,stmt_exit'. In other words, the binding of VAR should not escape the 'with' block. This would mean that for existing types that use __del__ for cleanup (e.g. files and sockets), then 'with open("file") as f' would automatically ensure closing under CPython (but other implementations would be allowed to wait for GC). In other words, I'm saying that this: with some_expr() as foo: # etc. should also be short for this (in the case where some_expr() has no __enter__ or __exit__ methods): foo = some_expr() try: # etc. finally: del foo And that could be a useful thing for many existing object types, without even updating them for PEP 34[0-9]. :) It wouldn't be *as* useful for non-CPython implementations, but presumably by the time those implementations catch up, more code will be out there with __enter__/__exit__ methods. Also, by allowing a default __enter__ to exist (that returns self), many objects need only implement an __exit__. (For example, I don't see a point to closed file objects raising an error when used in a 'with' block; if you're actually using the file you'll already get an error when you use its other methods, and if you're not actually using it, there's no point to the error, since close() is idempotent.) So, at the C API level, I'm thinking something like Py_EnterResource(ob), that returns ob if ob has no tp_resource_enter slot defined, otherwise it returns the result of calling the method. Similarly, some sort of Py_ExitResource() that guarantees an error return after invoking the tp_resource_exit slot (if any). Finally, note that this extension now makes 'with' seem more like 'with' in other languages, because it is now just a scoped variable definition, with hooks for the object being scoped to be notified about entry and exit from scope. It does mean that people encountering 'with some_expr()' (without an "as") may wonder about whether names inside the scope are somehow relative to 'some_expr', but it will probably become clear from context, especially via appropriate names. For example 'with self.__locked' might provide that extra bit of clarity beyond 'with self.__lock'. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com