Re: [Python-Dev] best place for an atomic file API
(MvL complained in the tracker issue about a lack of concrete use cases, but I think fixing race conditions when overwriting bytecode files in importlib and the existing distutils/packaging use cases cover that) I certainly agree that there are applications of atomic replace, and that the os module should expose the relevant platform APIs where available. I'm not so sure that atomic writes is a useful concept. I haven't seen a proposed implementation, yet, but I'm doubtful that truly ACID writes are possible unless the operating system supports transactions (which only Windows 7 does). Even if you are ignoring Isolation, Atomic already is a challenge: if you first write to a tempfile, then rename it, you may end up with a state tempfile (e.g. if the process is killed), and no rollback operation. So atomic write to me promises something that it likely can't deliver. OTOH, I still think that the promise isn't actually asked for in practice (not even when overwriting bytecode files) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Most users don't need a truly ACID write, but implement their own best-effort function. Instead of having a different implement in each project, Python can provide something better, especially when the OS provides low level function to implement such feature. Victor 2012/2/16 Martin v. Löwis mar...@v.loewis.de: I'm not so sure that atomic writes is a useful concept. I haven't seen a proposed implementation, yet, but I'm doubtful that truly ACID writes are possible unless the operating system supports transactions (which only Windows 7 does). Even if you are ignoring Isolation, Atomic already is a challenge: if you first write to a tempfile, then rename it, you may end up with a state tempfile (e.g. if the process is killed), and no rollback operation. So atomic write to me promises something that it likely can't deliver. OTOH, I still think that the promise isn't actually asked for in practice (not even when overwriting bytecode files) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Am 16.02.2012 10:54, schrieb Victor Stinner: Most users don't need a truly ACID write, but implement their own best-effort function. Instead of having a different implement in each project, Python can provide something better, especially when the OS provides low level function to implement such feature. It's then critical how this is named, IMO (and exactly what semantics it comprises). Calling it atomic when it is not is a mistake. Also notice that one user commented that that he already implemented something like this, and left out the issue of *permissions*. I found that interesting, since preserving permissions might indeed a requirement in a lot of in-place update use cases, but hasn't been considered in this discussion yet. So rather than providing a mechanism for atomic writes, I think providing a mechanism to update a file is what people might need. One way of providing this might be a u mode for open, which updates an existing file on close (unlike a, which appends, and unlike w, which truncates first). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Martin v. Löwis martin at v.loewis.de writes: One way of providing this might be a u mode for open, which updates an existing file on close (unlike a, which appends, and unlike w, which truncates first). Doesn't r+ cover this? Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
15.02.12 23:16, Charles-François Natali написав(ла): Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. For example, it could be used by importlib to avoid races when writting bytecode files (issues #13392, #13003, #13146), or more generally by any application that wants to make sure to end up with a consistent file even in face of crash (e.g. it seems that mercurial implemented their own version). What if target file is symlink? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] best place for an atomic file API
Hi,
Issue #8604 aims at adding an atomic file API to make it easier to
create/update files atomically, using rename() on POSIX systems and
MoveFileEx() on Windows (which are now available through
os.replace()). It would also use fsync() on POSIX to make sure data is
committed to disk.
For example, it could be used by importlib to avoid races when
writting bytecode files (issues #13392, #13003, #13146), or more
generally by any application that wants to make sure to end up with a
consistent file even in face of crash (e.g. it seems that mercurial
implemented their own version).
Basically the usage would be, e.g.:
with AtomicFile('foo') as f:
pickle.dump(obj, f)
or
with AtomicFile('foo') as f:
chunk = heavyCrunch()
f.write(chunk)
chunk = CrunchSomeMore()
f.write(chunk)
What would be the best place for a such a class?
_pyio, tempfile, or a new atomicfile
Cheers,
cf
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Charles-François Natali wrote: Hi, Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. [...] What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile shutil perhaps? As a user, that's the third place I look for file utilities, after builtin functions and os module. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
On Thu, Feb 16, 2012 at 9:29 AM, Steven D'Aprano st...@pearwood.info wrote: Charles-François Natali wrote: What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile shutil perhaps? As a user, that's the third place I look for file utilities, after builtin functions and os module. +1 for shutil from me. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Charles-François Natali neolo...@free.fr writes: Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. These make it quite OS-specific. […] What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile I would expect to find it within ‘os’ or submodules of ‘os’. -- \“We should be less concerned about adding years to life, and | `\ more about adding life to years.” —Arthur C. Clarke, 2001 | _o__) | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
On Wed, Feb 15, 2012 at 19:19, Ben Finney ben+pyt...@benfinney.id.au wrote: Charles-François Natali neolo...@free.fr writes: Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. These make it quite OS-specific. That'll happen when solving problems on different OSes. Do you propose a more platform agnostic solution? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
Brian Curtin br...@python.org writes: On Wed, Feb 15, 2012 at 19:19, Ben Finney ben+pyt...@benfinney.id.au wrote: Charles-François Natali neolo...@free.fr writes: […] using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. These make it quite OS-specific. That'll happen when solving problems on different OSes. Do you propose a more platform agnostic solution? No, I have no objection to that implementation. I'm pointing that out only because the nature of the functionality implies I'd expect to find it within the ‘os’ module hierarchy. -- \ “The man who is denied the opportunity of taking decisions of | `\ importance begins to regard as important the decisions he is | _o__)allowed to take.” —C. Northcote Parkinson | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] best place for an atomic file API
On Thu, Feb 16, 2012 at 11:49 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: No, I have no objection to that implementation. I'm pointing that out only because the nature of the functionality implies I'd expect to find it within the ‘os’ module hierarchy. The (very) rough rule of thumb is that the os module handles abstracting away cross-platform differences in implementation details, while the higher level shutil algorithms can be largely platform independent by using the shared abstractions in the os module layer. In this case, os.replace() is the cross platform abstraction, while the atomic file context manager is just a particular use case for that new feature. (MvL complained in the tracker issue about a lack of concrete use cases, but I think fixing race conditions when overwriting bytecode files in importlib and the existing distutils/packaging use cases cover that) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
