Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
Hi! > >>User wants to test for a file with name "foo.txt" > >> > >>* create "foo.txt~" (or whatever) > >>* write contents into "foo.txt~" > >>* rename "foo.txt~" to "foo.txt" > >> > >>Until rename is done, the file does not exists and is not complete. > >>You will potentially have a garbage file to clean up if the program > >>(or system) crashes, but that is not racy in a classic sense, right? > >Well. If people rsync from you, they will start fetching incomplete > >foo.txt~. Plus the garbage issue. > > That is not racy, just garbage (not trying to be pedantic, just > trying to understand). I can see that the "~" file is annoying, but > we have dealt with it for a *long* time :) Ok, so lets keep it at "~" is annoying :-). [But... I was wrong. openat(..., AT_UNLINKED) is not enough to solve this: we do not have flink() and it is not easily possible to link deleted file "back to life" from /proc/self/fd: pavel@amd:/tmp$ > delme pavel@amd:/tmp$ bash 3< delme & [2] 32667 [2]+ Stopped bash 3< delme pavel@amd:/tmp$ fg bash 3< delme pavel@amd:/tmp$ ls -al delme -rw-r--r-- 1 pavel pavel 0 Apr 1 01:36 delme pavel@amd:/tmp$ ls -al /proc/self/fd/3 lr-x-- 1 pavel pavel 64 Apr 1 01:37 /proc/self/fd/3 -> /tmp/delme pavel@amd:/tmp$ rm delme pavel@amd:/tmp$ ls -al /proc/self/fd/3 lr-x-- 1 pavel pavel 64 Apr 1 01:37 /proc/self/fd/3 -> /tmp/delme (deleted) pavel@amd:/tmp$ ln /proc/self/fd/3 delme2 ln: creating hard link `delme2' => `/proc/self/fd/3': Invalid cross-device link ] > >>This is more of a garbage clean up issue? > >Also. Plus sometimes you want temporary "file" that is > >deleted. Terminals use it for history, etc... > > There you would have a race, you can create a file and unlink it of > course and still write to it, but you would have a potential empty > file issue? Yes. openat(..., AT_UNLINKED) solves that -- you'll no longer get those files. (Not sure they'd be always empty. How do you ensure rm hits the disk? fsync() on parent directory? Sounds expensive.) Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On 03/31/2013 07:18 PM, Pavel Machek wrote: Hi! Take a look at how many actively used filesystems out there that have some variant of sillyrename(), and explain what you want to do in those cases. Well. Yes, there are non-unix filesystems around. You have to deal with silly files on them, and this will not be different. So this would be a local POSIX filesystem only solution to a problem that has yet to be formulated? Problem is "clasical create temp file then delete it" is racy. See the archives. That is useful & common operation. Which race are you concerned with exactly? User wants to test for a file with name "foo.txt" * create "foo.txt~" (or whatever) * write contents into "foo.txt~" * rename "foo.txt~" to "foo.txt" Until rename is done, the file does not exists and is not complete. You will potentially have a garbage file to clean up if the program (or system) crashes, but that is not racy in a classic sense, right? Well. If people rsync from you, they will start fetching incomplete foo.txt~. Plus the garbage issue. That is not racy, just garbage (not trying to be pedantic, just trying to understand). I can see that the "~" file is annoying, but we have dealt with it for a *long* time :) Until it has the right name (on either the source or target system for rsync), it is not the file you are looking for. This is more of a garbage clean up issue? Also. Plus sometimes you want temporary "file" that is deleted. Terminals use it for history, etc... There you would have a race, you can create a file and unlink it of course and still write to it, but you would have a potential empty file issue? Ric -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
Hi! > Take a look at how many actively used filesystems out there that have > some variant of sillyrename(), and explain what you want to do in those > cases. > >>>Well. Yes, there are non-unix filesystems around. You have to deal > >>>with silly files on them, and this will not be different. > >>So this would be a local POSIX filesystem only solution to a problem > >>that has yet to be formulated? > >Problem is "clasical create temp file then delete it" is racy. See the > >archives. That is useful & common operation. > > Which race are you concerned with exactly? > > User wants to test for a file with name "foo.txt" > > * create "foo.txt~" (or whatever) > * write contents into "foo.txt~" > * rename "foo.txt~" to "foo.txt" > > Until rename is done, the file does not exists and is not complete. > You will potentially have a garbage file to clean up if the program > (or system) crashes, but that is not racy in a classic sense, right? Well. If people rsync from you, they will start fetching incomplete foo.txt~. Plus the garbage issue. > This is more of a garbage clean up issue? Also. Plus sometimes you want temporary "file" that is deleted. Terminals use it for history, etc... Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On 03/31/2013 06:50 PM, Pavel Machek wrote:
On Sun 2013-03-31 18:44:53, Myklebust, Trond wrote:
On Sun, 2013-03-31 at 20:32 +0200, Pavel Machek wrote:
Hmm. open_deleted_file() will still need to get a directory... so it
will still need a path. Perhaps open("/foo/bar/mnt", O_DELETED) would
be acceptable interface?
...and what's the big plan to make this work on anything other than ext4 and
btrfs?
Deleted but open files are from original unix, so it should work on
anything unixy (minix, ext, ext2, ...).
minix, ext, ext2... are not under active development and haven't been
for more than a decade.
Take a look at how many actively used filesystems out there that have
some variant of sillyrename(), and explain what you want to do in those
cases.
Well. Yes, there are non-unix filesystems around. You have to deal
with silly files on them, and this will not be different.
So this would be a local POSIX filesystem only solution to a problem
that has yet to be formulated?
Problem is "clasical create temp file then delete it" is racy. See the
archives. That is useful & common operation.
Which race are you concerned with exactly?
User wants to test for a file with name "foo.txt"
* create "foo.txt~" (or whatever)
* write contents into "foo.txt~"
* rename "foo.txt~" to "foo.txt"
Until rename is done, the file does not exists and is not complete. You will
potentially have a garbage file to clean up if the program (or system) crashes,
but that is not racy in a classic sense, right?
This is more of a garbage clean up issue?
Regards,
Ric
Problem is "atomicaly create file at target location with guaranteed
right content". That's also in the archives. Looks useful if someone
does rsync from your directory.
Non-POSIX filesystems have problems handling deleted files, but that
was always the case. That's one of the reasons they are seldomly used
for root filesystems.
Pavel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On Sun 2013-03-31 18:44:53, Myklebust, Trond wrote:
> On Sun, 2013-03-31 at 20:32 +0200, Pavel Machek wrote:
> > > > > > Hmm. open_deleted_file() will still need to get a directory... so it
> > > > > > will still need a path. Perhaps open("/foo/bar/mnt", O_DELETED)
> > > > > > would
> > > > > > be acceptable interface?
> > > > >
> > > > > ...and what's the big plan to make this work on anything other than
> > > > > ext4 and btrfs?
> > > >
> > > > Deleted but open files are from original unix, so it should work on
> > > > anything unixy (minix, ext, ext2, ...).
> > >
> > > minix, ext, ext2... are not under active development and haven't been
> > > for more than a decade.
> > >
> > > Take a look at how many actively used filesystems out there that have
> > > some variant of sillyrename(), and explain what you want to do in those
> > > cases.
> >
> > Well. Yes, there are non-unix filesystems around. You have to deal
> > with silly files on them, and this will not be different.
>
> So this would be a local POSIX filesystem only solution to a problem
> that has yet to be formulated?
Problem is "clasical create temp file then delete it" is racy. See the
archives. That is useful & common operation.
Problem is "atomicaly create file at target location with guaranteed
right content". That's also in the archives. Looks useful if someone
does rsync from your directory.
Non-POSIX filesystems have problems handling deleted files, but that
was always the case. That's one of the reasons they are seldomly used
for root filesystems.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures)
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On Sun, 2013-03-31 at 20:32 +0200, Pavel Machek wrote:
> > > > > Hmm. open_deleted_file() will still need to get a directory... so it
> > > > > will still need a path. Perhaps open("/foo/bar/mnt", O_DELETED) would
> > > > > be acceptable interface?
> > > >
> > > > ...and what's the big plan to make this work on anything other than
> > > > ext4 and btrfs?
> > >
> > > Deleted but open files are from original unix, so it should work on
> > > anything unixy (minix, ext, ext2, ...).
> >
> > minix, ext, ext2... are not under active development and haven't been
> > for more than a decade.
> >
> > Take a look at how many actively used filesystems out there that have
> > some variant of sillyrename(), and explain what you want to do in those
> > cases.
>
> Well. Yes, there are non-unix filesystems around. You have to deal
> with silly files on them, and this will not be different.
So this would be a local POSIX filesystem only solution to a problem
that has yet to be formulated?
--
Trond Myklebust
Linux NFS client maintainer
NetApp
trond.mykleb...@netapp.com
www.netapp.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
> > > > Hmm. open_deleted_file() will still need to get a directory... so it
> > > > will still need a path. Perhaps open("/foo/bar/mnt", O_DELETED) would
> > > > be acceptable interface?
> > >
> > > ...and what's the big plan to make this work on anything other than ext4
> > > and btrfs?
> >
> > Deleted but open files are from original unix, so it should work on
> > anything unixy (minix, ext, ext2, ...).
>
> minix, ext, ext2... are not under active development and haven't been
> for more than a decade.
>
> Take a look at how many actively used filesystems out there that have
> some variant of sillyrename(), and explain what you want to do in those
> cases.
Well. Yes, there are non-unix filesystems around. You have to deal
with silly files on them, and this will not be different.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures)
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
Hmm. open_deleted_file() will still need to get a directory... so it will still need a path. Perhaps open(/foo/bar/mnt, O_DELETED) would be acceptable interface? ...and what's the big plan to make this work on anything other than ext4 and btrfs? Deleted but open files are from original unix, so it should work on anything unixy (minix, ext, ext2, ...). minix, ext, ext2... are not under active development and haven't been for more than a decade. Take a look at how many actively used filesystems out there that have some variant of sillyrename(), and explain what you want to do in those cases. Well. Yes, there are non-unix filesystems around. You have to deal with silly files on them, and this will not be different. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On Sun, 2013-03-31 at 20:32 +0200, Pavel Machek wrote: Hmm. open_deleted_file() will still need to get a directory... so it will still need a path. Perhaps open(/foo/bar/mnt, O_DELETED) would be acceptable interface? ...and what's the big plan to make this work on anything other than ext4 and btrfs? Deleted but open files are from original unix, so it should work on anything unixy (minix, ext, ext2, ...). minix, ext, ext2... are not under active development and haven't been for more than a decade. Take a look at how many actively used filesystems out there that have some variant of sillyrename(), and explain what you want to do in those cases. Well. Yes, there are non-unix filesystems around. You have to deal with silly files on them, and this will not be different. So this would be a local POSIX filesystem only solution to a problem that has yet to be formulated? -- Trond Myklebust Linux NFS client maintainer NetApp trond.mykleb...@netapp.com www.netapp.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On Sun 2013-03-31 18:44:53, Myklebust, Trond wrote: On Sun, 2013-03-31 at 20:32 +0200, Pavel Machek wrote: Hmm. open_deleted_file() will still need to get a directory... so it will still need a path. Perhaps open(/foo/bar/mnt, O_DELETED) would be acceptable interface? ...and what's the big plan to make this work on anything other than ext4 and btrfs? Deleted but open files are from original unix, so it should work on anything unixy (minix, ext, ext2, ...). minix, ext, ext2... are not under active development and haven't been for more than a decade. Take a look at how many actively used filesystems out there that have some variant of sillyrename(), and explain what you want to do in those cases. Well. Yes, there are non-unix filesystems around. You have to deal with silly files on them, and this will not be different. So this would be a local POSIX filesystem only solution to a problem that has yet to be formulated? Problem is clasical create temp file then delete it is racy. See the archives. That is useful common operation. Problem is atomicaly create file at target location with guaranteed right content. That's also in the archives. Looks useful if someone does rsync from your directory. Non-POSIX filesystems have problems handling deleted files, but that was always the case. That's one of the reasons they are seldomly used for root filesystems. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On 03/31/2013 06:50 PM, Pavel Machek wrote: On Sun 2013-03-31 18:44:53, Myklebust, Trond wrote: On Sun, 2013-03-31 at 20:32 +0200, Pavel Machek wrote: Hmm. open_deleted_file() will still need to get a directory... so it will still need a path. Perhaps open(/foo/bar/mnt, O_DELETED) would be acceptable interface? ...and what's the big plan to make this work on anything other than ext4 and btrfs? Deleted but open files are from original unix, so it should work on anything unixy (minix, ext, ext2, ...). minix, ext, ext2... are not under active development and haven't been for more than a decade. Take a look at how many actively used filesystems out there that have some variant of sillyrename(), and explain what you want to do in those cases. Well. Yes, there are non-unix filesystems around. You have to deal with silly files on them, and this will not be different. So this would be a local POSIX filesystem only solution to a problem that has yet to be formulated? Problem is clasical create temp file then delete it is racy. See the archives. That is useful common operation. Which race are you concerned with exactly? User wants to test for a file with name foo.txt * create foo.txt~ (or whatever) * write contents into foo.txt~ * rename foo.txt~ to foo.txt Until rename is done, the file does not exists and is not complete. You will potentially have a garbage file to clean up if the program (or system) crashes, but that is not racy in a classic sense, right? This is more of a garbage clean up issue? Regards, Ric Problem is atomicaly create file at target location with guaranteed right content. That's also in the archives. Looks useful if someone does rsync from your directory. Non-POSIX filesystems have problems handling deleted files, but that was always the case. That's one of the reasons they are seldomly used for root filesystems. Pavel -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
Hi! Take a look at how many actively used filesystems out there that have some variant of sillyrename(), and explain what you want to do in those cases. Well. Yes, there are non-unix filesystems around. You have to deal with silly files on them, and this will not be different. So this would be a local POSIX filesystem only solution to a problem that has yet to be formulated? Problem is clasical create temp file then delete it is racy. See the archives. That is useful common operation. Which race are you concerned with exactly? User wants to test for a file with name foo.txt * create foo.txt~ (or whatever) * write contents into foo.txt~ * rename foo.txt~ to foo.txt Until rename is done, the file does not exists and is not complete. You will potentially have a garbage file to clean up if the program (or system) crashes, but that is not racy in a classic sense, right? Well. If people rsync from you, they will start fetching incomplete foo.txt~. Plus the garbage issue. This is more of a garbage clean up issue? Also. Plus sometimes you want temporary file that is deleted. Terminals use it for history, etc... Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
On 03/31/2013 07:18 PM, Pavel Machek wrote: Hi! Take a look at how many actively used filesystems out there that have some variant of sillyrename(), and explain what you want to do in those cases. Well. Yes, there are non-unix filesystems around. You have to deal with silly files on them, and this will not be different. So this would be a local POSIX filesystem only solution to a problem that has yet to be formulated? Problem is clasical create temp file then delete it is racy. See the archives. That is useful common operation. Which race are you concerned with exactly? User wants to test for a file with name foo.txt * create foo.txt~ (or whatever) * write contents into foo.txt~ * rename foo.txt~ to foo.txt Until rename is done, the file does not exists and is not complete. You will potentially have a garbage file to clean up if the program (or system) crashes, but that is not racy in a classic sense, right? Well. If people rsync from you, they will start fetching incomplete foo.txt~. Plus the garbage issue. That is not racy, just garbage (not trying to be pedantic, just trying to understand). I can see that the ~ file is annoying, but we have dealt with it for a *long* time :) Until it has the right name (on either the source or target system for rsync), it is not the file you are looking for. This is more of a garbage clean up issue? Also. Plus sometimes you want temporary file that is deleted. Terminals use it for history, etc... There you would have a race, you can create a file and unlink it of course and still write to it, but you would have a potential empty file issue? Ric -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: openat(..., AT_UNLINKED) was Re: New copyfile system call - discuss before LSF?
Hi! User wants to test for a file with name foo.txt * create foo.txt~ (or whatever) * write contents into foo.txt~ * rename foo.txt~ to foo.txt Until rename is done, the file does not exists and is not complete. You will potentially have a garbage file to clean up if the program (or system) crashes, but that is not racy in a classic sense, right? Well. If people rsync from you, they will start fetching incomplete foo.txt~. Plus the garbage issue. That is not racy, just garbage (not trying to be pedantic, just trying to understand). I can see that the ~ file is annoying, but we have dealt with it for a *long* time :) Ok, so lets keep it at ~ is annoying :-). [But... I was wrong. openat(..., AT_UNLINKED) is not enough to solve this: we do not have flink() and it is not easily possible to link deleted file back to life from /proc/self/fd: pavel@amd:/tmp$ delme pavel@amd:/tmp$ bash 3 delme [2] 32667 [2]+ Stopped bash 3 delme pavel@amd:/tmp$ fg bash 3 delme pavel@amd:/tmp$ ls -al delme -rw-r--r-- 1 pavel pavel 0 Apr 1 01:36 delme pavel@amd:/tmp$ ls -al /proc/self/fd/3 lr-x-- 1 pavel pavel 64 Apr 1 01:37 /proc/self/fd/3 - /tmp/delme pavel@amd:/tmp$ rm delme pavel@amd:/tmp$ ls -al /proc/self/fd/3 lr-x-- 1 pavel pavel 64 Apr 1 01:37 /proc/self/fd/3 - /tmp/delme (deleted) pavel@amd:/tmp$ ln /proc/self/fd/3 delme2 ln: creating hard link `delme2' = `/proc/self/fd/3': Invalid cross-device link ] This is more of a garbage clean up issue? Also. Plus sometimes you want temporary file that is deleted. Terminals use it for history, etc... There you would have a race, you can create a file and unlink it of course and still write to it, but you would have a potential empty file issue? Yes. openat(..., AT_UNLINKED) solves that -- you'll no longer get those files. (Not sure they'd be always empty. How do you ensure rm hits the disk? fsync() on parent directory? Sounds expensive.) Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
