On Mar 15, 2009, at 3:25 PM, Greg Ewing wrote:
with renaming_file("blarg.txt", "w") as f:
...
By chance during the weekend I actually wrote something like that:
from __future__ import with_statement
import os
import codecs
import shutil
import tempfile
from contextlib import contextma
Greg Ewing wrote:
> What might make more sense is a context manager,
> e.g.
>
> with renaming_file("blarg.txt", "w") as f:
> ...
As you were describing the problems with "rename on close", I actually
immediately thought of the oft-repeated db transaction commit/rollback
example from PEP 343
>
>
> > Ok. In that use case, however, it is completely irrelevant whether the
> > tempfile module calls fsync. After it has generated the non-conflicting
> > filename, it's done.
>
> I agree, but my comment was that it would be nice if better fsync
> support (if added) could be done in such a way
Nick Coghlan wrote:
It actually wouldn't be a bad place to put a "create a temporary file
and rename it to when closing it" helper class.
I'm not sure it would be a good idea to make that
behaviour automatic on closing. If anything goes
wrong while writing the file, you *don't* want the
renam
Op vrijdag 13-03-2009 om 12:28 uur [tijdzone +0300], schreef Oleg
Broytmann:
>Ext4 is not the only FS with delayed allocation.
Of course not, even ext3 has delayed allocation (even if 5 sec vs. 2 min
makes the disaster window a bit smaller).
The real problem seems to be that ext4 re-orders th
Zvezdan Petkovic wrote:
> Of course, the above are C functions. I don't think that Python
> programming is immune from such security considerations either.
The tempfile module exposes the same functionality (and uses mkstemp()
to create its filenames). It has also had features added over the year
On Fri, Mar 13, 2009 at 07:31:21PM +0100, "Martin v. Löwis" wrote:
> > Think about the security implications of a file name that is in advance
> > known to an attacker as well as the fact that the said file will replace
> > an *important* system file.
>
> You should always use O_EXCL in that case.
On Mar 13, 2009, at 2:31 PM, Martin v. Löwis wrote:
Think about the security implications of a file name that is in
advance known to an attacker as well as the fact that the said file
will replace an *important* system file.
You should always use O_EXCL in that case. Relying on random name
> Think about the security implications of a file name that is in advance
> known to an attacker as well as the fact that the said file will replace
> an *important* system file.
You should always use O_EXCL in that case. Relying on random name will
be a severe security threat to the application.
On Fri, Mar 13, 2009 at 12:28:07PM +0300, Oleg Broytmann wrote:
> On Thu, Mar 12, 2009 at 10:14:41PM -0600, Adam Olsen wrote:
> > Yet the ext4
> > developers didn't see it that way, so it was sacrificed to new
> > performance improvements (delayed allocation).
>
>Ext4 is not the only FS with d
On Mar 12, 2009, at 3:15 PM, Martin v. Löwis wrote:
You still wouldn't use the tempfile module in that case. Instead, you
would create a regular file, with the name base on the name of the
important file.
If the file is *really* important, you actually want to use a
temporary, randomly chose
Martin v. Löwis wrote:
>> auto-delete is one of the nice features of tempfile. Another feature
>> which is entirely appropriate to this usage, though, though, is creation
>> of a non-conflicting filename.
>
> Ok. In that use case, however, it is completely irrelevant whether the
> tempfile module
On Thu, Mar 12, 2009 at 10:14:41PM -0600, Adam Olsen wrote:
> Yet the ext4
> developers didn't see it that way, so it was sacrificed to new
> performance improvements (delayed allocation).
Ext4 is not the only FS with delayed allocation. New XFS has it, btrfs
will have it. Don't know about othe
On Tue, Mar 10, 2009 at 2:11 PM, Christian Heimes wrote:
> Multiple blogs and news sites are swamped with a discussion about ext4
> and KDE 4.0. Theodore Ts'o - the developer of ext4 - explains the issue
> at
> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54.
>
>
> Pyt
Martin v. Löwis wrote:
>> auto-delete is one of the nice features of tempfile. Another feature
>> which is entirely appropriate to this usage, though, though, is creation
>> of a non-conflicting filename.
>
> Ok. In that use case, however, it is completely irrelevant whether the
> tempfile module
> auto-delete is one of the nice features of tempfile. Another feature
> which is entirely appropriate to this usage, though, though, is creation
> of a non-conflicting filename.
Ok. In that use case, however, it is completely irrelevant whether the
tempfile module calls fsync. After it has gener
Martin v. Löwis wrote:
The sync is necessary to ensure that the data is written to the disk
before the old file overwrites the new filename.
>>> You still wouldn't use the tempfile module in that case. Instead, you
>>> would create a regular file, with the name base on the name of the
>>>
>>> The sync is necessary to ensure that the data is written to the disk
>>> before the old file overwrites the new filename.
>> You still wouldn't use the tempfile module in that case. Instead, you
>> would create a regular file, with the name base on the name of the
>> important file.
>>
> Uhm...
Martin v. Löwis wrote:
>> Something that doesn't require deterministicly named tempfiles was Ted
>> T'so's explanation linked to earlier.
>>
>> read data from important file
>> modify data
>> create tempfile
>> write data to tempfile
>> *sync tempfile to disk*
>> mv tempfile to filename of importan
> Something that doesn't require deterministicly named tempfiles was Ted
> T'so's explanation linked to earlier.
>
> read data from important file
> modify data
> create tempfile
> write data to tempfile
> *sync tempfile to disk*
> mv tempfile to filename of important file
>
> The sync is necessa
Antoine Pitrou wrote:
> Steven D'Aprano pearwood.info> writes:
>> It depends on what you mean by "temporary".
>>
>> Applications like OpenOffice can sometimes recover from an application
>> crash or even a systems crash and give you the opportunity to restore
>> the temporary files that were lef
Steven D'Aprano pearwood.info> writes:
>
> It depends on what you mean by "temporary".
>
> Applications like OpenOffice can sometimes recover from an application
> crash or even a systems crash and give you the opportunity to restore
> the temporary files that were left lying around.
For such
On Thu, 12 Mar 2009 01:03:13 pm Antoine Pitrou wrote:
> Nick Coghlan gmail.com> writes:
> > The tempfile module would be another example.
>
> Do you really need your temporary files to survive system crashes? ;)
It depends on what you mean by "temporary".
Applications like OpenOffice can sometim
Nick Coghlan gmail.com> writes:
>
> On the performance side... the overhead from fsync() itself is going to
> dwarf the CPU overhead of going through a wrapper class.
The significant overhead is not in calling sync() or flush() or close(), but in
calling methods which are supposed to be fast (re
On Mar 11, 2009, at 22:43 , Cameron Simpson wrote:
On 11Mar2009 10:09, Joachim K?nig wrote:
Guido van Rossum wrote:
On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes
wrote:
[...]
https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54
.
[...]
If I understand the post
Antoine Pitrou wrote:
> Nick Coghlan gmail.com> writes:
>> The tempfile module would be another example.
>
> Do you really need your temporary files to survive system crashes? ;)
No, but they need to provide the full file API. If we add a sync()
method to file objects, that becomes part of the "
Nick Coghlan gmail.com> writes:
>
> The tempfile module would be another example.
Do you really need your temporary files to survive system crashes? ;)
> For that reason, I think Steven's idea of a filetools module which
> provided context managers and the like that wrapped *existing* file-like
Greg Ewing wrote:
> Antoine Pitrou wrote:
>
>> What do you mean? open() doesn't allow you to wrap other file objects.
>
> I'm talking about things like GzipFile that take a
> filename and mode, open the file and then wrap the
> file object.
The tempfile module would be another example.
For that
Antoine Pitrou wrote:
What do you mean? open() doesn't allow you to wrap other file objects.
I'm talking about things like GzipFile that take a
filename and mode, open the file and then wrap the
file object.
--
Greg
___
Python-Dev mailing list
Pytho
Greg Ewing canterbury.ac.nz> writes:
>
> I like this, because it doesn't expand the signature that
> file-like objects need to support. If you're wrapping
> another file object you just need to pass on the mode
> string and it will all work.
What do you mean? open() doesn't allow you to wrap oth
2009/3/11 Greg Ewing :
> Lie Ryan wrote:
>
>> I actually prefer strings. Just like 'w' or 'r' in open().
>>
>> Or why not add "f" "c" as modes?
>>
>> open('file.txt', 'wf')
>
> I like this, because it doesn't expand the signature that
> file-like objects need to support. If you're wrapping
> anothe
Martin v. Löwis wrote:
That should be implement by passing O_SYNC on open, rather than
explicitly calling fsync.
On platforms which have it (MacOSX doesn't seem to,
according to the man page).
This is another good reason to put these things in the
mode string.
--
Greg
___
Lie Ryan wrote:
I actually prefer strings. Just like 'w' or 'r' in open().
Or why not add "f" "c" as modes?
open('file.txt', 'wf')
I like this, because it doesn't expand the signature that
file-like objects need to support. If you're wrapping
another file object you just need to pass on the
On Thu, 12 Mar 2009 01:21:25 am Antoine Pitrou wrote:
> Christian Heimes cheimes.de> writes:
> > In my initial proposal one and a half hour earlier I suggested
> > 'sync()' as the name of the method and 'synced' as the name of the
> > flag that forces a fsync() call during the close operation.
>
>
On 11Mar2009 10:09, Joachim K?nig wrote:
> Guido van Rossum wrote:
>> On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes wrote:
>>> [...]
>>> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54.
>>> [...]
>> If I understand the post properly, it's up to the app to call fsy
Barry Warsaw wrote:
Of course, a careful *nix application can ensure that the file owners
and mod bits are set the way it needs them to be set. A convenience
function might be useful though.
A specialised function would also provide a place for
dealing with platform-specific extensions, su
Antoine Pitrou:
> It depends on what you call "ACLs". It does copy the chmod permission bits.
Access Control Lists are fine grained permissions. Perhaps you
want to allow Sam to read a file and for Ted to both read and write
it. These permissions should not need to be reset every time you
mod
> Maybe it would make more sense for "synced" to force fsync() on each
> flush, not only on close. I'm not sure how useful it is, but that's
> what "synced" would imply to me.
That should be implement by passing O_SYNC on open, rather than
explicitly calling fsync.
Regards,
Martin
__
> This is especially true given Windows has recently introduced a
> transactional API for NTFS. Although the tone is - err - gushing - it
> (a) should give some information on what is available, and (b) was high
> on my google search list
>
> http://msdn.microsoft.com/en-us/magazine/cc163388.asp
Antoine Pitrou wrote:
Eric Smith trueblade.com> writes:
Why wouldn't sync just be an optional argument to close(), at least for
the "sync_on_close" case?
It wouldn't work with the "with" statement.
Well, that is a good reason, then!
___
Python-De
Scott Dial wrote:
Aahz wrote:
On Wed, Mar 11, 2009, Antoine Pitrou wrote:
After Hrvoje's message, let me rephrase my suggestion. Let's instead allow:
open(..., sync_on="close")
open(..., sync_on="flush")
with a default of None meaning no implicit syncs.
That looks good, though I'd prefe
Eric Smith trueblade.com> writes:
>
> Why wouldn't sync just be an optional argument to close(), at least for
> the "sync_on_close" case?
It wouldn't work with the "with" statement.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.pytho
Antoine Pitrou wrote:
I think your "synced" flag is too vague. Some applications may need the file to
be synced on close(), but some others may need it to be synced at regular
intervals, or after each write(), etc.
Why wouldn't sync just be an optional argument to close(), at least for
the "sy
On Wed, Mar 11, 2009, Scott Dial wrote:
> Aahz wrote:
>> On Wed, Mar 11, 2009, Antoine Pitrou wrote:
>>> After Hrvoje's message, let me rephrase my suggestion. Let's instead allow:
>>>open(..., sync_on="close")
>>>open(..., sync_on="flush")
>>>
>>> with a default of None meaning no implicit
Aahz wrote:
> On Wed, Mar 11, 2009, Antoine Pitrou wrote:
>> After Hrvoje's message, let me rephrase my suggestion. Let's instead allow:
>>open(..., sync_on="close")
>>open(..., sync_on="flush")
>>
>> with a default of None meaning no implicit syncs.
>
> That looks good, though I'd prefer
On Wed, Mar 11, 2009, Antoine Pitrou wrote:
>
> After Hrvoje's message, let me rephrase my suggestion. Let's instead allow:
>open(..., sync_on="close")
>open(..., sync_on="flush")
>
> with a default of None meaning no implicit syncs.
That looks good, though I'd prefer using named constant
Christian Heimes cheimes.de> writes:
>
> And sync_on="flush" implies sync_on="close"?
close() implies flush(), so by construction yes.
> Your suggestion sounds like
> the right way to me!
I'm glad I brought something constructive to the discussion :-))
___
Antoine Pitrou schrieb:
> After Hrvoje's message, let me rephrase my suggestion. Let's instead allow:
>open(..., sync_on="close")
>open(..., sync_on="flush")
>
> with a default of None meaning no implicit syncs.
And sync_on="flush" implies sync_on="close"? Your suggestion sounds like
the
After Hrvoje's message, let me rephrase my suggestion. Let's instead allow:
open(..., sync_on="close")
open(..., sync_on="flush")
with a default of None meaning no implicit syncs.
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.o
Christian Heimes cheimes.de> writes:
>
> In my initial proposal one and a half hour earlier I suggested 'sync()'
> as the name of the method and 'synced' as the name of the flag that
> forces a fsync() call during the close operation.
I think your "synced" flag is too vague. Some applications ma
Christian Heimes wrote:
Guido van Rossum wrote:
Let's not think too Unix-specific. If we add such an API it should do
something on Windows too -- the app shouldn't have to test for the
presence of the API. (And thus the API probably shouldn't be called
fsync.)
In my initial proposal one and a
Guido van Rossum wrote:
> Let's not think too Unix-specific. If we add such an API it should do
> something on Windows too -- the app shouldn't have to test for the
> presence of the API. (And thus the API probably shouldn't be called
> fsync.)
In my initial proposal one and a half hour earlier I
Oleg Broytmann phd.pp.ru> writes:
>
>That's easy to fix - only copy ownership if the effective user id == 0.
But errors should not pass silently. If the user intended the function to copy
ownership information and the function fails to do so, it should raise an
exception.
Having implicit sp
On 11/03/2009 1:55 PM, Guido van Rossum wrote:
On Tue, Mar 10, 2009 at 7:45 PM, Christian Heimes wrote:
Antoine Pitrou wrote:
Christian Heimes cheimes.de> writes:
...
Let's not think too Unix-specific. If we add such an API it should do
something on Windows too -- the app shouldn't have to
On Wed, Mar 11, 2009 at 11:56:13AM +, Antoine Pitrou wrote:
> Oleg Broytmann phd.pp.ru> writes:
> >Only root can change file ownership - and yes, there are scripts that
> > run with root privileges, so why not copy?
>
> Because the new function would then be useless for non-root scripts
Oleg Broytmann phd.pp.ru> writes:
>
>Only root can change file ownership - and yes, there are scripts that
> run with root privileges, so why not copy?
Because the new function would then be useless for non-root scripts, and
encouraging people to run their scripts as root would be rather bad
Christian Heimes cheimes.de> writes:
>
> It's more than an object oriented convenience. fsync() takes a file
> descriptor as argument. Therefore I assume fsync() only syncs the data
> to disk that was written to the file descriptor.
Ok, I agree that a .sync() method makes sense.
__
On Wed, Mar 11, 2009 at 11:43:33AM +, Antoine Pitrou wrote:
> As for owner and group, I think there is a very good reason that it doesn't
> copy
> them: under Linux, only root can change these properties.
Only root can change file ownership - and yes, there are scripts that
run with root p
Neil Hodgson gmail.com> writes:
>
>shutil.copystat does not copy over the owner, group or ACLs.
It depends on what you call "ACLs". It does copy the chmod permission bits.
As for owner and group, I think there is a very good reason that it doesn't copy
them: under Linux, only root can change
Joachim König wrote:
To me, the flaw seem to be in the close() call (of the operating
system). I'd expect the data to be
in a persistent state once the close() returns.
I wouldn't, because that would mean that every cp -r would effectively
do an fsync() for each individual file it copies, whi
Antoine Pitrou:
> How about shutil.copystat()?
shutil.copystat does not copy over the owner, group or ACLs.
Modeling a copymetadata method on copystat would provide an easy to
understand API and should be implementable on Windows and POSIX.
Reading the OS X documentation shows a set of low
Guido van Rossum wrote:
On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes wrote:
[...]
https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54.
[...]
If I understand the post properly, it's up to the app to call fsync(),
and it's only necessary when you're doing on
>> We already have os.fsync() and os.fdatasync(). Should the sync() (and
>> datasync()?) method be added as an object-oriented convenience?
>
> It's more than an object oriented convenience. fsync() takes a file
> descriptor as argument. Therefore I assume fsync() only syncs the data
> to disk tha
A.M. Kuchling wrote:
With zipfile, you could at least access the .fp attribute
to sync it (though is the .fp documented as part of the interface?).
For this one, I'd like to add the sync as a method (so that Zip-inside-
Zip is eventually possible). In fact, a sync on an exposed writable
On 11Mar2009 02:20, Antoine Pitrou wrote:
| Christian Heimes cheimes.de> writes:
| > I agree with you, fsync() shouldn't be called by default. I didn't plan
| > on adding fsync() calls all over our code. However I like to suggest a
| > file.sync() method and a synced flag for files to make the jo
On 10Mar2009 22:14, A.M. Kuchling wrote:
| On Wed, Mar 11, 2009 at 11:31:52AM +1100, Cameron Simpson wrote:
| > On 10Mar2009 18:09, A.M. Kuchling wrote:
| > | The mailbox module tries to be careful and always fsync() before
| > | closing files, because mail messages are pretty important.
| >
| >
On Tue, Mar 10, 2009 at 7:45 PM, Christian Heimes wrote:
> Antoine Pitrou wrote:
>> Christian Heimes cheimes.de> writes:
>>> I agree with you, fsync() shouldn't be called by default. I didn't plan
>>> on adding fsync() calls all over our code. However I like to suggest a
>>> file.sync() method an
Antoine Pitrou wrote:
> Christian Heimes cheimes.de> writes:
>> I agree with you, fsync() shouldn't be called by default. I didn't plan
>> on adding fsync() calls all over our code. However I like to suggest a
>> file.sync() method and a synced flag for files to make the job of
>> application deve
Christian Heimes cheimes.de> writes:
>
> I agree with you, fsync() shouldn't be called by default. I didn't plan
> on adding fsync() calls all over our code. However I like to suggest a
> file.sync() method and a synced flag for files to make the job of
> application developers easier.
We alread
On Wed, Mar 11, 2009 at 11:31:52AM +1100, Cameron Simpson wrote:
> On 10Mar2009 18:09, A.M. Kuchling wrote:
> | The mailbox module tries to be careful and always fsync() before
> | closing files, because mail messages are pretty important.
>
> Can it be turned off? I hadn't realised this.
No, th
Guido van Rossum wrote:
> If I understand the post properly, it's up to the app to call fsync(),
> and it's only necessary when you're doing one of the rename dances, or
> updating a file in place. Basically, as he explains, fsync() is a very
> heavyweight operation; I'm against calling it by defau
On 10Mar2009 18:09, A.M. Kuchling wrote:
| On Tue, Mar 10, 2009 at 09:11:38PM +0100, Christian Heimes wrote:
| > Python's file type doesn't use fsync() and be the victim of the very
| > same issue, too. Should we do anything about it?
IMHO, beyond _offering_ an fsync method, no.
| The mailbox mo
Neil Hodgson gmail.com> writes:
>
> What would be useful is a simple, generic
> way in Python to copy all the appropriate metadata (ownership, ACLs,
> ...) to another file so the temporary-and-rename technique could be
> used.
How about shutil.copystat()?
__
On Tue, Mar 10, 2009 at 09:11:38PM +0100, Christian Heimes wrote:
> Python's file type doesn't use fsync() and be the victim of the very
> same issue, too. Should we do anything about it?
The mailbox module tries to be careful and always fsync() before
closing files, because mail messages are pret
> If I understand the post properly, it's up to the app to call fsync(),
Correct.
> and it's only necessary when you're doing one of the rename dances, or
> updating a file in place.
No. It's in general necessary when you want to be sure that the data is
on disk, even if the power is lost. So e
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Mar 10, 2009, at 4:46 PM, Neil Hodgson wrote:
The technique advocated by Theodore Ts'o (save to temporary then
rename) discards metadata. What would be useful is a simple, generic
way in Python to copy all the appropriate metadata (ownership, A
On Tue, Mar 10, 2009 at 1:46 PM, Neil Hodgson wrote:
> The technique advocated by Theodore Ts'o (save to temporary then
> rename) discards metadata. What would be useful is a simple, generic
> way in Python to copy all the appropriate metadata (ownership, ACLs,
> ...) to another file so the temp
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Mar 10, 2009, at 4:23 PM, Guido van Rossum wrote:
On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes
wrote:
Multiple blogs and news sites are swamped with a discussion about
ext4
and KDE 4.0. Theodore Ts'o - the developer of ext4 - explains th
The technique advocated by Theodore Ts'o (save to temporary then
rename) discards metadata. What would be useful is a simple, generic
way in Python to copy all the appropriate metadata (ownership, ACLs,
...) to another file so the temporary-and-rename technique could be
used.
On Windows, the
On Tue, Mar 10, 2009 at 1:11 PM, Christian Heimes wrote:
> Multiple blogs and news sites are swamped with a discussion about ext4
> and KDE 4.0. Theodore Ts'o - the developer of ext4 - explains the issue
> at
> https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54.
>
>
> Pyt
Multiple blogs and news sites are swamped with a discussion about ext4
and KDE 4.0. Theodore Ts'o - the developer of ext4 - explains the issue
at
https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/54.
Python's file type doesn't use fsync() and be the victim of the very
same
81 matches
Mail list logo
