Re: File not closed on exception
On 20 Okt, 21:13, "Gabriel Genellina" wrote: > En Tue, 20 Oct 2009 04:47:02 -0300, arve.knud...@gmail.com > escribió: > > > On 20 Okt, 09:40, "Gabriel Genellina" wrote: > >> En Tue, 20 Oct 2009 03:23:49 -0300, arve.knud...@gmail.com > >> escribió: > >> > I agree, but like I said, I've been told that this (implicit closing > >> > of files) is the correct style by more merited Python developers, so > >> > that made me think I was probably wrong .. > > >> Then tell those "more merited Python developers" that they're wrong, > >> and that the right way to ensure a file is closed when you're done with > >> it is to use a `with` statement (or a try/finally block in old Python > >> releases) > > > Easier said than done :) In any case, I now have this discussion as a > > useful reference in the future. Thanks! > > If this thread is not enough, you can ask them to read the official Python > tutorial: > > "It is good practice to use the with keyword when dealing with file > objects. This has the advantage that the file is properly closed after its > suite finishes, even if an exception is raised on the way. It is also much > shorter than writing equivalent try-finally blocks." > > http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects Perhaps the general attitude has changed now that the "with" keyword makes it so easy anyway (unless one needs to support older Pythons of course). Arve -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
En Tue, 20 Oct 2009 04:47:02 -0300, arve.knud...@gmail.com escribió: On 20 Okt, 09:40, "Gabriel Genellina" wrote: En Tue, 20 Oct 2009 03:23:49 -0300, arve.knud...@gmail.com escribió: > I agree, but like I said, I've been told that this (implicit closing > of files) is the correct style by more merited Python developers, so > that made me think I was probably wrong .. Then tell those "more merited Python developers" that they're wrong, and that the right way to ensure a file is closed when you're done with it is to use a `with` statement (or a try/finally block in old Python releases) Easier said than done :) In any case, I now have this discussion as a useful reference in the future. Thanks! If this thread is not enough, you can ask them to read the official Python tutorial: "It is good practice to use the with keyword when dealing with file objects. This has the advantage that the file is properly closed after its suite finishes, even if an exception is raised on the way. It is also much shorter than writing equivalent try-finally blocks." http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
On 20 Okt, 16:00, Bruno Desthuilliers wrote: > What's your problem with the with ??? No problem whatsoever, but I believe I wrote this utility function before the keyword was available, and it might be good to support older Python versions. > But anyway : explicitely releasing resources such as files, network > connections etc is of course the RightThing(tm), except eventually in > one-shot throwaway scripts. > > > However, when providing a patch for a > > high-profile opensource Python project I was scolded for going to such > > lengths, as the prescribed style was to just open files and let them > > be closed implicitly. > > Err... Care to name the project ? I hope it's not one I ever advertized :( I'm not going to name the project, but it is incidentally used by the Python project itself :) Arve -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
arve.knud...@gmail.com a écrit : On Oct 19, 4:14 pm, Grant Edwards wrote: On 2009-10-19, arve.knud...@gmail.com wrote: I thought that file objects were supposed to be garbage-collected and automatically closed once they go out of scope, At some point after they go out of scope, they will be. Eventually. Exactly when is an implementation detail. at least that's what I've been told by more merited Python programmers. I'm also quite sure that this is quite a common assumption in various programs, AFAICT, the file descriptor associated to the file object will be freed when the CPython process will finish - which implies you're using the CPython interpreter. This doesn't mean the file is garanteed to be closed _at the point where you're trying to os.remove() it_. And as I said, this is a CPython implementation detail - not a language specification. Jython or IronPython (or any other implementation) may not work that way. If your program relies on the assumption that some particular object will be garbage-collected between points A and B, then that's a bug in your program. If you depend on the fact that some object has been delted, then "del" it. If you depend on the fact that a file is closed, then close it. Personally I am against that assumption, and prefer a utility function which reads the file and automatically closes it in a "finally" block (in lieu of the "with" keyword). What's your problem with the with ??? But anyway : explicitely releasing resources such as files, network connections etc is of course the RightThing(tm), except eventually in one-shot throwaway scripts. However, when providing a patch for a high-profile opensource Python project I was scolded for going to such lengths, as the prescribed style was to just open files and let them be closed implicitly. Err... Care to name the project ? I hope it's not one I ever advertized :( -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
arve.knud...@gmail.com wrote:
On Oct 19, 3:48 pm, Ethan Furman wrote:
arve.knud...@gmail.com wrote:
Hi
I thought that file objects were supposed to be garbage-collected and
automatically closed once they go out of scope, at least that's what
I've been told by more merited Python programmers. I'm also quite sure
that this is quite a common assumption in various programs, at least
given what opensource code I've seen in my time. However, the
following script doesn't work on Windows, since the file is still open
when I try to remove it:
import os.path
def create():
f = file("tmp", "w")
raise Exception
try: create()
finally:
os.remove("tmp")
So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?
Thanks!
Arve
When an exception is raised, the entire stack frame at that location
(which includes local vars) is saved in the exception traceback. Since
the objects are still alive, they are not GC'ed. That is why this is
better:
def create():
f = file("tmp", "w")
try:
do_stuff_that_raises_exception
finally:
os.remove("tmp")
~Ethan~
Why should this work? If I replace "do_stuff_that_raises_exception"
with "raise Exception", it fails in the same way, since the file is
open. Maybe you forgot "f.close()"? In any case, thanks for explaining
that the traceback keeps the object alive, that explains the issue.
Arve
Indeed. That should have been f.close() in the finally block. My
apologies.
~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
arve.knud...@gmail.com wrote: > I agree, but like I said, I've been told that this (implicit closing > of files) is the correct style by more merited Python developers, so > that made me think I was probably wrong .. It would be nice. The trouble is that CPython is not the only Python. Jython, for example, uses the Java Runtime Environment for its virtual machine. JRE doesn't have reference-counts, so Jython can't close files immediately after the last reference ends. It seems guaranteed object cleanup would lock Python out of too many possible platforms. Mel. -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
arve.knud...@gmail.com wrote:
> On Oct 19, 3:48 pm, Ethan Furman wrote:
>> arve.knud...@gmail.com wrote:
[...]
>>> def create():
>>> f = file("tmp", "w")
>>> raise Exception
>>>
>>> try:
>>> create()
>>> finally:
>>> os.remove("tmp")
>>>
[...]
>> When an exception is raised, the entire stack frame at that location
>> (which includes local vars) is saved in the exception traceback. [...]
>> this is better:
>>
>> def create():
>> f = file("tmp", "w")
>> try:
>> do_stuff_that_raises_exception
>> finally:
>> os.remove("tmp")
[...]
> Why should this work? If I replace "do_stuff_that_raises_exception"
> with "raise Exception", it fails in the same way, since the file is
> open. Maybe you forgot "f.close()"?
I was puzzled by the same, but too lazy to try or ask. Anyhow, I think that
if you replaced the 'os.remove("tmp")' with 'f.close()', then the calling
function can remain the same as you wrote. This is basically the same as
when using the new-style "with", as mentioned by Gabriel.
Uli
--
Sator Laser GmbH
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
--
http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
On 20 Okt, 09:40, "Gabriel Genellina" wrote: > En Tue, 20 Oct 2009 03:23:49 -0300, arve.knud...@gmail.com > escribió: > > > > > > > On Oct 19, 5:56 pm, "Gabriel Genellina" > > wrote: > >> En Mon, 19 Oct 2009 09:45:49 -0200, arve.knud...@gmail.com > >> escribió: > > >> > I thought that file objects were supposed to be garbage-collected and > >> > automatically closed once they go out of scope, at least that's what > >> > I've been told by more merited Python programmers. > > >> When an object holds references to external resources that must be > >> freed, > >> this is not a good idea. Being explicit with the resource deallocation > >> is > >> much better than relying on object destruction sometime in the future... > > > I agree, but like I said, I've been told that this (implicit closing > > of files) is the correct style by more merited Python developers, so > > that made me think I was probably wrong .. > > Then tell those "more merited Python developers" that they're wrong, and > that the right way to ensure a file is closed when you're done with it is > to use a `with` statement (or a try/finally block in old Python releases) Easier said than done :) In any case, I now have this discussion as a useful reference in the future. Thanks! Arve -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
En Tue, 20 Oct 2009 03:23:49 -0300, arve.knud...@gmail.com escribió: On Oct 19, 5:56 pm, "Gabriel Genellina" wrote: En Mon, 19 Oct 2009 09:45:49 -0200, arve.knud...@gmail.com escribió: > I thought that file objects were supposed to be garbage-collected and > automatically closed once they go out of scope, at least that's what > I've been told by more merited Python programmers. When an object holds references to external resources that must be freed, this is not a good idea. Being explicit with the resource deallocation is much better than relying on object destruction sometime in the future... I agree, but like I said, I've been told that this (implicit closing of files) is the correct style by more merited Python developers, so that made me think I was probably wrong .. Then tell those "more merited Python developers" that they're wrong, and that the right way to ensure a file is closed when you're done with it is to use a `with` statement (or a try/finally block in old Python releases) -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
On Oct 19, 5:56 pm, "Gabriel Genellina" wrote: > En Mon, 19 Oct 2009 09:45:49 -0200, arve.knud...@gmail.com > escribió: > > > I thought that file objects were supposed to be garbage-collected and > > automatically closed once they go out of scope, at least that's what > > I've been told by more merited Python programmers. > > An object (any object) is destroyed as soon as the last reference to the > it is removed. A local variable holds a reference to the file object; it > that is the ONLY reference, the file object will be destroyed when the > variable goes out of scope, yes. > Note that: > - there might be more references to the object > - garbage collection is a separate subject; objects are reference-counted, > zero=>kaputt, the GC has no say on this. GC is only used to break cycles > (a->b, b->a) that would prevent the objects to reach 0 references. > - this behavior is specific of CPython > > > I'm also quite sure > > that this is quite a common assumption in various programs, at least > > given what opensource code I've seen in my time. > > When an object holds references to external resources that must be freed, > this is not a good idea. Being explicit with the resource deallocation is > much better than relying on object destruction sometime in the future... I agree, but like I said, I've been told that this (implicit closing of files) is the correct style by more merited Python developers, so that made me think I was probably wrong .. Arve -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
On Oct 19, 4:14 pm, Grant Edwards wrote: > On 2009-10-19, arve.knud...@gmail.com wrote: > > > I thought that file objects were supposed to be > > garbage-collected and automatically closed once they go out of > > scope, > > At some point after they go out of scope, they will be. > Eventually. Exactly when is an implementation detail. > > > at least that's what I've been told by more merited Python > > programmers. I'm also quite sure that this is quite a common > > assumption in various programs, > > If your program relies on the assumption that some particular > object will be garbage-collected between points A and B, then > that's a bug in your program. If you depend on the fact that > some object has been delted, then "del" it. If you depend on > the fact that a file is closed, then close it. Personally I am against that assumption, and prefer a utility function which reads the file and automatically closes it in a "finally" block (in lieu of the "with" keyword). However, when providing a patch for a high-profile opensource Python project I was scolded for going to such lengths, as the prescribed style was to just open files and let them be closed implicitly. Also, the problem may arise when I call a function in a 3rd party library, that it opens files which I then can't delete upon an exception from within said function. Actually, something like that did happen and spurred my original question, but fortunately a reference to the file was kept in the 3rd party object I was operating on, so I was able to free it in a "finally" block. Arve -- http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
On Oct 19, 3:48 pm, Ethan Furman wrote:
> arve.knud...@gmail.com wrote:
> > Hi
>
> > I thought that file objects were supposed to be garbage-collected and
> > automatically closed once they go out of scope, at least that's what
> > I've been told by more merited Python programmers. I'm also quite sure
> > that this is quite a common assumption in various programs, at least
> > given what opensource code I've seen in my time. However, the
> > following script doesn't work on Windows, since the file is still open
> > when I try to remove it:
>
> > import os.path
>
> > def create():
> > f = file("tmp", "w")
> > raise Exception
>
> > try: create()
> > finally:
> > os.remove("tmp")
>
> > So, what's the deal exactly, is the file supposed to be garbage-
> > collected (and closed) at the end of create?
>
> > Thanks!
> > Arve
>
> When an exception is raised, the entire stack frame at that location
> (which includes local vars) is saved in the exception traceback. Since
> the objects are still alive, they are not GC'ed. That is why this is
> better:
>
> def create():
> f = file("tmp", "w")
> try:
> do_stuff_that_raises_exception
> finally:
> os.remove("tmp")
>
> ~Ethan~
Why should this work? If I replace "do_stuff_that_raises_exception"
with "raise Exception", it fails in the same way, since the file is
open. Maybe you forgot "f.close()"? In any case, thanks for explaining
that the traceback keeps the object alive, that explains the issue.
Arve
--
http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
En Mon, 19 Oct 2009 09:45:49 -0200, arve.knud...@gmail.com
escribió:
I thought that file objects were supposed to be garbage-collected and
automatically closed once they go out of scope, at least that's what
I've been told by more merited Python programmers.
An object (any object) is destroyed as soon as the last reference to the
it is removed. A local variable holds a reference to the file object; it
that is the ONLY reference, the file object will be destroyed when the
variable goes out of scope, yes.
Note that:
- there might be more references to the object
- garbage collection is a separate subject; objects are reference-counted,
zero=>kaputt, the GC has no say on this. GC is only used to break cycles
(a->b, b->a) that would prevent the objects to reach 0 references.
- this behavior is specific of CPython
I'm also quite sure
that this is quite a common assumption in various programs, at least
given what opensource code I've seen in my time.
When an object holds references to external resources that must be freed,
this is not a good idea. Being explicit with the resource deallocation is
much better than relying on object destruction sometime in the future...
However, the
following script doesn't work on Windows, since the file is still open
when I try to remove it:
import os.path
def create():
f = file("tmp", "w")
raise Exception
try: create()
finally:
os.remove("tmp")
So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?
The object does not go out of scope because there is an additional
reference: the exception traceback holds a reference to all execution
frames, and each frame holds a reference to its local variables. So "f"
is still alive. This is quite good for a debugger, or for logging
purposes, as one can inspect the values of each and every variable along
the frame chain. But until exception processing is finished, the "f"
variable is alive and the "tmp" file is open.
How to deal with this depends on your use case. I don't know what can I
modify on your small example and still being representative of your actual
problem. The reccomended way to process a file uses a with statement:
def create():
with open("tmp", "w") as f:
# do something with the file
raise Exception
This way the file is closed when leaving the with statement (either
normally or because of an exception). And, if the file doesn't exist or
access is denied, the open() call doesn't success either. In any case, it
never remains open.
--
Gabriel Genellina
--
http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
On 2009-10-19, arve.knud...@gmail.com wrote:
> I thought that file objects were supposed to be
> garbage-collected and automatically closed once they go out of
> scope,
At some point after they go out of scope, they will be.
Eventually. Exactly when is an implementation detail.
> at least that's what I've been told by more merited Python
> programmers. I'm also quite sure that this is quite a common
> assumption in various programs,
If your program relies on the assumption that some particular
object will be garbage-collected between points A and B, then
that's a bug in your program. If you depend on the fact that
some object has been delted, then "del" it. If you depend on
the fact that a file is closed, then close it.
> at least given what opensource code I've seen in my time.
> However, the following script doesn't work on Windows, since
> the file is still open when I try to remove it:
>
> import os.path
>
> def create():
> f = file("tmp", "w")
> raise Exception
>
> try: create()
> finally:
> os.remove("tmp")
>
> So, what's the deal exactly, is the file supposed to be garbage-
> collected (and closed) at the end of create?
Nothing is "supposed" to be garbage-collected. An object _may_
be garbage collected after some point.
--
Grant
--
http://mail.python.org/mailman/listinfo/python-list
Re: File not closed on exception
arve.knud...@gmail.com wrote:
Hi
I thought that file objects were supposed to be garbage-collected and
automatically closed once they go out of scope, at least that's what
I've been told by more merited Python programmers. I'm also quite sure
that this is quite a common assumption in various programs, at least
given what opensource code I've seen in my time. However, the
following script doesn't work on Windows, since the file is still open
when I try to remove it:
import os.path
def create():
f = file("tmp", "w")
raise Exception
try: create()
finally:
os.remove("tmp")
So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?
Thanks!
Arve
When an exception is raised, the entire stack frame at that location
(which includes local vars) is saved in the exception traceback. Since
the objects are still alive, they are not GC'ed. That is why this is
better:
def create():
f = file("tmp", "w")
try:
do_stuff_that_raises_exception
finally:
os.remove("tmp")
~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list
File not closed on exception
Hi
I thought that file objects were supposed to be garbage-collected and
automatically closed once they go out of scope, at least that's what
I've been told by more merited Python programmers. I'm also quite sure
that this is quite a common assumption in various programs, at least
given what opensource code I've seen in my time. However, the
following script doesn't work on Windows, since the file is still open
when I try to remove it:
import os.path
def create():
f = file("tmp", "w")
raise Exception
try: create()
finally:
os.remove("tmp")
So, what's the deal exactly, is the file supposed to be garbage-
collected (and closed) at the end of create?
Thanks!
Arve
--
http://mail.python.org/mailman/listinfo/python-list
