[issue17852] Built-in module _io can lose data from buffered files at exit

Nathaniel Smith added the comment: > I attempted to implement my weakref idea (i.e. raw file keeps a weakref to > the buffered file, calls flush before the raw file gets closed). That > doesn't work either because the GC clears the weakref before calling __del__. This may be

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: Using atexit is not the solution because the data can be lost even while the program is running, not just at interpreter shutdown. The problem occurs if the buffered file object and the underlying file object are both part of a

[issue17852] Built-in module _io can lose data from buffered files at exit

Arusekk added the comment: Since the issue seems to have been active lately, may I suggest my view on solving it. One solution that comes to my mind is to keep a weak reference to the file, and to register an atexit function per file (and to unregister it when the file is

[issue17852] Built-in module _io can lose data from buffered files at exit

Antoine Pitrou added the comment: Here's another idea: have a new type field (a tp_something) that exposes the GC priority for this type. It could be an integer between -8 and 7, for example (0 being the default). Then tp_finalize would be called in sorted priority order.

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: Yeah, I think you are correct. Currently files not part of reference cycles get properly flushed and closed by the reference counter. Implementing my "buffer_register_flush" patch would cause files to be closed only by the cyclic

[issue17852] Built-in module _io can lose data from buffered files at exit

Antoine Pitrou added the comment: I think it would be quite disruptive to create a reference cycle each time open() is called. It may also break user scripts. -- ___ Python tracker

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: Welp, another day another attempt. As mentioned in the PR 4847, atexit is not the answer. If the raw/buffered file pair are part of a reference cycle and the GC cleans it before atexit runs, then the buffered data can get lost. I

[issue17852] Built-in module _io can lose data from buffered files at exit

Nikolaus Rath added the comment: Given the apparent difficulties getting this right, how about not attempting to implicitly flush buffers in the finalizer at all? This means scripts relying on this will break, but in contrast to the current behavior they will break

[issue17852] Built-in module _io can lose data from buffered files at exit

Change by Nitish : -- nosy: +nitishch ___ Python tracker ___ ___ Python-bugs-list

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: Using reversed chronological order would work in 99% of the cases probably but then you would have that rare case where it didn't work. So, I'm not too keen on that approach. I think this is a classic problem with finalization and

[issue17852] Built-in module _io can lose data from buffered files at exit

Serhiy Storchaka added the comment: I'm not well known with GC, but if it be possible to call destructors in the reversed chronological order, this could help with this issue. -- ___ Python tracker

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: In the process of trying to write a test for this, I now realize that PR 4847 is not really a fix. If the underlying file gets closed during an earlier gc.collect() and not during shutdown, the extra flushing step is not going to

[issue17852] Built-in module _io can lose data from buffered files at exit

Change by Armin Rigo : -- nosy: -arigo ___ Python tracker ___ ___

[issue17852] Built-in module _io can lose data from buffered files at exit

Antoine Pitrou added the comment: The reason Python 2 did well here is simply that Python 2 had a single Python object (the file object) for each actual file. Python 3 has several of them (the raw IO object, the buffered IO object, possibly the text IO wrapper), and so

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: Attached is a script that triggers the non-flushing behaviour for me. I don't think it is reliable since it depends on the order that FileIO AND BufferedWriter are finalized when the gc finds them in a reference cycle. BTW, it is

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: I created a new PR which uses the atexit module instead of using _Py_PyAtExit. I think registering in io.py is okay. I see that atexit is now implemented in C. Rather than registering in io.py, we could create a C API to register

[issue17852] Built-in module _io can lose data from buffered files at exit

Change by Neil Schemenauer : -- pull_requests: +4735 stage: needs patch -> patch review ___ Python tracker ___

[issue17852] Built-in module _io can lose data from buffered files at exit

STINNER Victor added the comment: New changeset 317def9fdb29893df1ab380d396fcdd2eafe0588 by Victor Stinner (Antoine Pitrou) in branch 'master': bpo-17852: Revert incorrect fix based on misunderstanding of _Py_PyAtExit() semantics (#4826)

[issue17852] Built-in module _io can lose data from buffered files at exit

Antoine Pitrou added the comment: I just discovered that the fix is incorrect. See PR #4826 for reversion and a quick explanation. -- resolution: fixed -> stage: resolved -> needs patch status: closed -> open ___ Python tracker

[issue17852] Built-in module _io can lose data from buffered files at exit

Change by Antoine Pitrou : -- pull_requests: +4718 ___ Python tracker ___ ___

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: Yes, my bad. I thought that accepting the pull would close the bug. -- resolution: -> fixed stage: backport needed -> resolved status: open -> closed ___ Python tracker

[issue17852] Built-in module _io can lose data from buffered files at exit

STINNER Victor added the comment: Neil pushed the commit 0a1ff24acfc15d8c7f2dc41000a6f3d9a31e7480. What is the status of this issue? Can we now close it? -- ___ Python tracker

[issue17852] Built-in module _io can lose data from buffered files at exit

Change by Berker Peksag : -- stage: needs patch -> backport needed ___ Python tracker ___

[issue17852] Built-in module _io can lose data from buffered files at exit

Neil Schemenauer added the comment: New changeset 0a1ff24acfc15d8c7f2dc41000a6f3d9a31e7480 by Neil Schemenauer in branch 'master': bpo-17852: Maintain a list of BufferedWriter objects. Flush them on exit. (#3372)

[issue17852] Built-in module _io can lose data from buffered files at exit

Changes by Neil Schemenauer : -- pull_requests: +3382, 3383 ___ Python tracker ___

[issue17852] Built-in module _io can lose data from buffered files at exit

Changes by Neil Schemenauer : -- pull_requests: +3382 ___ Python tracker ___ ___

[issue17852] Built-in module _io can lose data from buffered files at exit

Changes by Raymond Hettinger : -- title: Built-in module _io can loose data from buffered files at exit -> Built-in module _io can lose data from buffered files at exit ___ Python tracker