Re: reiser4 plugins
On Monday July 11, [EMAIL PROTECTED] wrote: Stefan Smietanowski wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Why not implement it inside the directory containg the file ? Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo This should be suit both camps I'd think? You still need to figure out the parent of foo, which isn't necessarily easy, especially considering that even if we store a link to the parent, it will be inside the metadata. Then you have a chicken-and-egg situation. Both camps seem to want to allow easy access to the metadata of a file, whether we're given that file as an inode or as a pathname. That's why I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a file by name, and sometimes by inode, but either way, I should be able to get its metadata. Well, it's not really 'as an inode or as a pathname'. It is 'as an open file descriptor or as a path name' which is an important difference. Maybe it is worth repeating Al Viro's suggestion at this point. I don't have a reference but the idea was basically that if you open /foo and get filedescriptor N, then /proc/self/fds/N-meta is a directory which contains all the meta stuff for /foo. Then it is trivial to get the 'meta' stuff given a filedescriptor and if you have a pathname, you can always get yourself a filedescriptor. Then to allow cat /home/friend/foo/.meta.../perms you write a little .so which redefines open, stat, and a few others to scan for .meta... in the pathname and to the necessary magic, and export LD_PRELOAD=/that/.so Finally you write a killer app which fundamentally relies on this functionality, get everyone addicted to it, and *then* (and only then) start trying to get this functionality into the kernel. i.e. prototype new semantics in userspace. NeilBrown
Re: reiser4 plugins
Neil Brown wrote: Maybe it is worth repeating Al Viro's suggestion at this point. I don't have a reference but the idea was basically that if you open /foo and get filedescriptor N, then /proc/self/fds/N-meta is a directory which contains all the meta stuff for /foo. Then it is trivial to get the 'meta' stuff given a filedescriptor and if you have a pathname, you can always get yourself a filedescriptor. This sound like it might be cute, but filedescriptors are too heavy weight for stat data accesses in quantity. In general, the whole file handle paradigm is too heavy for lightweight files. Hans
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Neil Brown wrote: On Monday July 11, [EMAIL PROTECTED] wrote: Stefan Smietanowski wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Why not implement it inside the directory containg the file ? Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo This should be suit both camps I'd think? You still need to figure out the parent of foo, which isn't necessarily easy, especially considering that even if we store a link to the parent, it will be inside the metadata. Then you have a chicken-and-egg situation. Both camps seem to want to allow easy access to the metadata of a file, whether we're given that file as an inode or as a pathname. That's why I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a file by name, and sometimes by inode, but either way, I should be able to get its metadata. Well, it's not really 'as an inode or as a pathname'. It is 'as an open file descriptor or as a path name' which is an important difference. Maybe it is worth repeating Al Viro's suggestion at this point. I don't have a reference but the idea was basically that if you open /foo and get filedescriptor N, then /proc/self/fds/N-meta How am I supposed to get there with a shell script? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQtROqXgHNmZLgCUhAQLRcg/+I9PWSmFXRwKtj7pnEeMXOCjiTo6GQE3O 61fjH3f6aL9Ydkip4eXum3S14cioiU9bzC11GA5kRIM+W1DKcYex1dIpivrtF9Ht Rvozn9x2TP5tacDmSfqRJXvAB+xTRtZOu+M/rDjXdLsriDJDA0AdyDH8Yo/8WMbU 6i1DWzLTO0vHS3kEb/8oqgBj7sQ63sS/4KVszBx6+bN0KOikXbORDu6efKjC9w21 3DZPnBG0B03smhdCygd0j0Zmh0JEaZHfuFgNK1ZmRzipbvvUBDtdKY5MJ6f4pHnA GBO8ybsXp9qxNQr6DNenF/wbAT6n3dMyE/AWuql+qx3iumSwx/Prh7xDAhBZBMXp Oin7hOa1i583cdju4ErSBPaciRzumGluY6gbFvVA8Yva+IjPxxjPtfLwalK11cH1 k4oQO5Par1W0TmMOpc0PQ/bEeEHHxUcn1ToeJa4NYJWIiBe+UHMb/AyI4hKjSIkt Kp0wrCPBjRfuBCHXXL89bWZoSeSFkN8fAyOxBV928naxxr8e+vCPUX1/H3ca7UsB Nxg0Vzt4V9tz4xCw4QAy810Uya8/HSm3aVhqEzjHKBoKboHrMVDJvxRQxfkqQcnC 4jIFYPBdHgGw7OONyhfbgTPLIm1OCNPpcRkS4aidHqg0DkDU50h6zFQkhG5Xwh5Z x5REgxbqD+A= =FGm4 -END PGP SIGNATURE-
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hans Reiser wrote: Horst von Brand wrote: Hans Reiser [EMAIL PROTECTED] wrote: Stefan Smietanowski wrote: I think ... and .meta both serve as a logical delimiter. However some programs implement their own ... which would make it clash with them. Naturally if some program created a directory called .meta we're equally screwed. I chose '' (four dots) because it clashes with less, not three dots. Is this some kind of My dots are more than yours contest?! /None/ of them is safe. .meta, ..., , .this.has.five.dots. are all perfectly legal file (or directory) names, POSIXly. If any one of them won't do, none will. And, conversely, if any one of them makes sense, they all do. Well, with the exception that some of them have never, ever been used before, and that's what we should be aiming for. No conflicts with existing programs, then new programs can be written to avoid conflicting with us. There is a long history of encroaching namespaces by creating new keywords in CS, and it is a survivable problem. Agreed, but not one to be taken too lightly. Clearcase (the version control filesystem) is an excellent example. They have special meanings for @ and some other common characters, and you have to learn how to escape those characters if you use them in Clearcase. I like using @, though. Without having to escape it. That's why we're trying to find something that people won't actually touch, especially since if we design it right, this will be the last delimiter introduced at the fs/vfs level. It works, it makes users happy, in practice it is far less of a problem than one might think. I think there were two or three times I had to remember how to escape things in 2 years of using it. Better to spend one's mind looking for bugs instead of this issue. .if bugs were seen as such a big deal. I think it's far easier to get into the kernel with something ludicrously buggy than something which actually changes fundamental behavior. That is, you can put in an FS which actually corrupts data (such as the old NTFS write support), so long as it doesn't break POSIX, or cause other weird restrictions like No files named 'metas' Now, if we can decide that we don't care about being in the vanilla kernel, then we can just call it .metas or lost+found or whatever and get to work on bug fixes and other much-needed features such as a repacker. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQtRQvXgHNmZLgCUhAQJXWBAAlZggJQGNi2lEZh0MAqnz+rNVT1JYcY/b adHgvVOZZNiJw2GmVjGLIiG5cqSqML1//f0+4XOxjYED2rbfOwBJiDqcq/0IsKPp 5JJflJV2uWkEZukJ+oA5mspfSeKof2N5egBoD1Ije079VBKXdjoN5Kprkbe4ZYZ2 +Yu+w3FpdcaGvSqVlTRHPWsS0je4z8ieh0u6ql+HNR4ze/hKwMw4nrb2sARW9NOQ EZ8Ot5NDhVaxz9Rj72rLVuQrUZEF8b0ihkpmzTauVV/nysEGi33SPqYTF7nYGBnM 5mVY63uXG4zxmThMDpn+J/iofA/hS1fd1bY9tCgF1ZAPu9HrCTnVzKaTYoOq+ciD sY0m7HEc49JfaiyZ/SJGH02WUH3JqLQAVGQftEkqAQLyYdVRbzdBHUVUR2i8hX2M ofFLM6DGJgFK784PfO0sjNboQT5ay4B8js8NltdfytsOVwDzjMST7dZAWcnMgTZU jrMCJuMYeYh8468fy1C9pCMUPahVXvmF4O5Xazt7m9HvV4lxRJIUDZsaR0Volg2K OAQIO3rtT9heK/+/PaUuXX8RYwLQUlhdUmdHfdtjRiLtKRcbw/fYBOqjkgXZfOfy RzyBgD463sVCVAE8qMbAVnnHNGKZ25tyRTYITiTZ2kfnIeURJawj2SZVmn5ezGQG xRnI3+Ir5e0= =moA6 -END PGP SIGNATURE-
Re: reiser4 plugins
David Masover wrote: That's why we're trying to find something that people won't actually touch, especially since if we design it right, this will be the last delimiter introduced at the fs/vfs level. Uh, no, there needs to be about a dozen or so more. But not this year.
Re: reiser4 plugins
On Tuesday July 12, [EMAIL PROTECTED] wrote:
Maybe it is worth repeating Al Viro's suggestion at this point. I
don't have a reference but the idea was basically that if you open
/foo and get filedescriptor N, then
/proc/self/fds/N-meta
How am I supposed to get there with a shell script?
function get_meta()
{
var=$1
file=$2
meta=$3
val=`cat /proc/self/fd/3-meta/$meta $file`
eval var=\$val
}
then
get_meta varname /home/foo/bar username
will read the 'username' meta-file of 'home/foo/bar' and place it in
varname.
Is that what you wanted?
NeilBrown
Re: reiser4 plugins
On Tuesday July 12, [EMAIL PROTECTED] wrote: Neil Brown wrote: Maybe it is worth repeating Al Viro's suggestion at this point. I don't have a reference but the idea was basically that if you open /foo and get filedescriptor N, then /proc/self/fds/N-meta is a directory which contains all the meta stuff for /foo. Then it is trivial to get the 'meta' stuff given a filedescriptor and if you have a pathname, you can always get yourself a filedescriptor. This sound like it might be cute, but filedescriptors are too heavy weight for stat data accesses in quantity. In general, the whole file handle paradigm is too heavy for lightweight files. That may well be true, but is completely orthogonal to filesystem name semantics. If you find file descriptors too slow, come up with an alternate (I suspect you have in the reiser4 syscall, but I haven't looked at that yet), implement it in the VFS, and show the world benchmarks of real-world applications that go faster with this new interface. I doubt that you would then have a great deal of trouble in getting the interface accepted (some trouble of course as you will need to convince a few people, but numbers speak quite loudly). I suspect that there might need to be a new internal interface into filesystems, and filesystems which don't provide that will not get the same speed benefit, but that is perfectly acceptable. NeilBrown
Re: reiser4 plugins
Neil Brown wrote: On Tuesday July 12, [EMAIL PROTECTED] wrote: Neil Brown wrote: Maybe it is worth repeating Al Viro's suggestion at this point. I don't have a reference but the idea was basically that if you open /foo and get filedescriptor N, then /proc/self/fds/N-meta is a directory which contains all the meta stuff for /foo. Then it is trivial to get the 'meta' stuff given a filedescriptor and if you have a pathname, you can always get yourself a filedescriptor. This sound like it might be cute, but filedescriptors are too heavy weight for stat data accesses in quantity. In general, the whole file handle paradigm is too heavy for lightweight files. That may well be true, but is completely orthogonal to filesystem name semantics. If you find file descriptors too slow, come up with an alternate (I suspect you have in the reiser4 syscall, but I haven't looked at that yet), implement it in the VFS, and show the world benchmarks of real-world applications that go faster with this new interface. I doubt that you would then have a great deal of trouble in getting the interface accepted (some trouble of course as you will need to convince a few people, but numbers speak quite loudly). I suspect that there might need to be a new internal interface into filesystems, and filesystems which don't provide that will not get the same speed benefit, but that is perfectly acceptable. NeilBrown We need time, and then we can demonstrate sys_reiser4, it is not ready for showing yet. Hans
Re: reiser4 plugins
David Masover [EMAIL PROTECTED] wrote: Hans Reiser wrote: Horst von Brand wrote: Hans Reiser [EMAIL PROTECTED] wrote: Stefan Smietanowski wrote: [...] Better to spend one's mind looking for bugs instead of this issue. .if bugs were seen as such a big deal. I think it's far easier to get into the kernel with something ludicrously buggy than something which actually changes fundamental behavior. Wonder why [Fixing bugs in the $FOO driver or the $BAR filesystem is /easy/, fixing bugs in fundamental behaviour changes is /extremely hard/.] That is, you can put in an FS which actually corrupts data (such as the old NTFS write support), so long as it doesn't break POSIX, or cause other weird restrictions like No files named 'metas' Because that kind of problems are isolated. If you introduce a change that affects /all/ filesystems, and that change later on has unfixable bugs, or fundamental design issues, it is /a lot/ of work. Now, if we can decide that we don't care about being in the vanilla kernel, then we can just call it .metas or lost+found or whatever and get to work on bug fixes and other much-needed features such as a repacker. Great! -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
Hans Reiser wrote: David Masover wrote: That's why we're trying to find something that people won't actually touch, especially since if we design it right, this will be the last delimiter introduced at the fs/vfs level. Uh, no, there needs to be about a dozen or so more. Where? From what I (vaguely) remember of the future-vision paper, having the meta delimiter lets us do everything else from inside the metas. We can certainly add delimiters to stuff in a meta-dir... -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005
Re: reiser4 plugins
Stefan Smietanowski wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Why not implement it inside the directory containg the file ? Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo This should be suit both camps I'd think? You still need to figure out the parent of foo, which isn't necessarily easy, especially considering that even if we store a link to the parent, it will be inside the metadata. Then you have a chicken-and-egg situation. Both camps seem to want to allow easy access to the metadata of a file, whether we're given that file as an inode or as a pathname. That's why I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a file by name, and sometimes by inode, but either way, I should be able to get its metadata. I mean, editing something is easy and you don't have to know how to navigate /meta But then you have to know how to navigate .meta, and more importantly, how to find .meta and you don't have the clash of files vs metadata (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of the dir stesmi ?). So, how do I get metadata of a directory? If the metadata for /home/me is in /home/.meta/me, and the metadata for /home is in /.meta/home, then where is the metadata for / ? /home/stesmi/foo - dir /home/stesmi/.meta/foo - dir containing all metadata /home/stesmi/.meta/foo/attrib - some attribute called attrib /home/stesmi/foo/bar - file /home/stesmi/foo/.meta/bar - dir containg all metadata /home/stesmi/foo/.meta/bar/attrib - some attribute called attrib [...] If this has already been taken up, I must've missed it. It looks a lot like how I suggested we resolve the ambiguity within /meta/vfs, only I called it '...' instead of '.meta'. Either way, the major challenge to this method is not so much whether it'd work, but how to choose a suitable delimiter? The delimiter must be standard if we're going to have apps develop for it, and it must not be used in the name of any existing file or directory. I had a long essay here on how '.meta' breaks every recursive operation on directories (rm -rf, tar, mkisofs), but I deleted it when I remembered that I played around with the alpha/beta reiser4 which had a 'metas' dir that did the same thing, but was hidden from the normal directory listing. 'cd metas' worked, 'ls metas' worked, but 'ls' showed no directory called 'metas'. I don't *think* there are any apps that will break if you tell them to open a path that doesn't exist in a directory listing. That is, typing 'vim /home/metas/acl' should always let me edit the acl on /home, and vim should never complain about how /home/metas doesn't show up in /home. A program would have to be pretty retarded to fail on something like that. But, we have to support some pretty retarded programs. That is why I like /meta -- the default view is a completely POSIX-compliant tree that works with tar, and even the /meta view is POSIX-compliant, even if it'd be a bad idea to tar it. Then we don't have to worry at all about stupid programs. How much should we be worrying about this particular brand of stupidity? And what's a good delimiter? -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi. Why not implement it inside the directory containg the file ? Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo This should be suit both camps I'd think? You still need to figure out the parent of foo, which isn't necessarily easy, especially considering that even if we store a link to the parent, it will be inside the metadata. Then you have a chicken-and-egg situation. Either you're describing another problem than I'm thinking of or /home/stesmi/.meta/foo 's file will always be /home/stesmi/foo . $dir/.meta/$file and $dir/$file . Or do you mean something else when you say parent ? Do you mean that if you know $dir/.meta/$foo then you don't know the meta of the parent? Won't $dir/../.meta/$end-of-path/.meta solve that? /home/stesmi/foo - file /home/stesmi/.meta/foo - meta of foo /home/stesmi/ - parentdir /home/.meta/stesmi - meta of parentdir /home/stesmi/../.meta/stesmi - also meta of parentdir Or is your problem that you don't know stesmi ? Ie you can't figure out what the current dir name is ? So you can't construct /home/stesmi/../stesmi because you don't know the name of the dir? In that case do : /home/stesmi/.meta/. - meta of parent. Naturally same as /home/.meta/stesmi And if then .. is mirrored as well is a secondary discussion. /home/stesmi/.meta/. - meta of stesmi /home/stesmi/.meta/.. - meta of home /home/stesmi/.meta/foo - meta of file foo Both camps seem to want to allow easy access to the metadata of a file, whether we're given that file as an inode or as a pathname. That's why I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a file by name, and sometimes by inode, but either way, I should be able to get its metadata. I mean, editing something is easy and you don't have to know how to navigate /meta But then you have to know how to navigate .meta, and more importantly, how to find .meta The biggest problem in my eyes is exportability. and you don't have the clash of files vs metadata (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of the dir stesmi ?). So, how do I get metadata of a directory? If the metadata for /home/me is in /home/.meta/me, and the metadata for /home is in /.meta/home, then where is the metadata for / ? /.meta/. (look above) The . is also inside .meta /home/stesmi/foo - dir /home/stesmi/.meta/foo - dir containing all metadata /home/stesmi/.meta/foo/attrib - some attribute called attrib /home/stesmi/foo/bar - file /home/stesmi/foo/.meta/bar - dir containg all metadata /home/stesmi/foo/.meta/bar/attrib - some attribute called attrib [...] If this has already been taken up, I must've missed it. It looks a lot like how I suggested we resolve the ambiguity within /meta/vfs, only I called it '...' instead of '.meta'. Ah. When I read the talk about ... I mistook it for something else. I'm sorry. Either way, the major challenge to this method is not so much whether it'd work, but how to choose a suitable delimiter? The delimiter must be standard if we're going to have apps develop for it, and it must not be used in the name of any existing file or directory. I think ... and .meta both serve as a logical delimiter. However some programs implement their own ... which would make it clash with them. Naturally if some program created a directory called .meta we're equally screwed. I had a long essay here on how '.meta' breaks every recursive operation on directories (rm -rf, tar, mkisofs), but I deleted it when I remembered that I played around with the alpha/beta reiser4 which had a 'metas' dir that did the same thing, but was hidden from the normal directory listing. 'cd metas' worked, 'ls metas' worked, but 'ls' showed no directory called 'metas'. I don't *think* there are any apps that will break if you tell them to open a path that doesn't exist in a directory listing. That is, typing 'vim /home/metas/acl' should always let me edit the acl on /home, and vim should never complain about how /home/metas doesn't show up in /home. A program would have to be pretty retarded to fail on something like that. But, we have to support some pretty retarded programs. That is why I like /meta -- the default view is a completely POSIX-compliant tree that works with tar, and even the /meta view is POSIX-compliant, even if it'd be a bad idea to tar it. Then we don't have to worry at all about stupid programs. How much should we be worrying about this particular brand of stupidity? And what's a good delimiter? Very good points. I don't like it being seperate from the dir though so I'm more inclined to support having the metas close to the file itself (/home/stesmi/.meta/) instead of seperate from (/meta). // Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (MingW32) iD8DBQFC0tygBrn2kJu9P78RAmBYAJ9YVeAacrREuBFrpGz9Ov5STo6dXwCgxrFi d2BN/T+//CfIZXPCUN9uhC4= =6SfT -END PGP
Re: reiser4 plugins
Stefan Smietanowski wrote: I think ... and .meta both serve as a logical delimiter. However some programs implement their own ... which would make it clash with them. Naturally if some program created a directory called .meta we're equally screwed. I chose '' (four dots) because it clashes with less, not three dots.
Re: reiser4 plugins
On Sun, 10 Jul 2005 02:00:49 +0200, Stefan Smietanowski [EMAIL PROTECTED] said: So basically if I write a program that works in both Gnome and KDE I should (according to your description) implement my own VFS that will use the Gnome or KDE VFS that will then use the OS VFS. Either that, or use a whole lot of #ifdefs, or hope that GNOME and KDE agree on a common VFS, or don't use their VFS and just use the basic OS calls and lose the functionality (and portability) of the VFSes, or pick one VFS and hope that the other users can adapt. GNOME and KDE already have their own VFS. I am not asking for anything new. If you don't like the idea of a VFS at that level, take it up with the GNOME and KDE people. Is it only me finding that a little silly? I mean, if I am to have the same functionality under neither Gnome nor VFS and they don't support something I need I _NEED_ a vfs so that my program is so totally independent on anything at all. My program calling My VFS which calls KDE/Gnome's VFS which calls the OS VFS will be slowe than just calling the VFS immidiately - I do hope you can see that. Of course. It's probably slower than arranging the bits on the hard drive directly, and hand-coding everything in assembly. But there's always a performance price to pay for maintaining the programmer's sanity. There's always a price to pay when writing cross-platform stuff. -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
Hans Reiser [EMAIL PROTECTED] wrote: Stefan Smietanowski wrote: I think ... and .meta both serve as a logical delimiter. However some programs implement their own ... which would make it clash with them. Naturally if some program created a directory called .meta we're equally screwed. I chose '' (four dots) because it clashes with less, not three dots. Is this some kind of My dots are more than yours contest?! /None/ of them is safe. .meta, ..., , .this.has.five.dots. are all perfectly legal file (or directory) names, POSIXly. If any one of them won't do, none will. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
David Masover [EMAIL PROTECTED] wrote: [...] Both camps seem to want to allow easy access to the metadata of a file, whether we're given that file as an inode or as a pathname. That's why I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a file by name, and sometimes by inode, but either way, I should be able to get its metadata. You should /never/ give access just by inode, as that makes it very easy to bypass security given by directory permissions. I mean, editing something is easy and you don't have to know how to navigate /meta But then you have to know how to navigate .meta, and more importantly, how to find .meta And what the heck the format of the metadata is today/for this particular file. [...] Either way, the major challenge to this method is not so much whether it'd work, but how to choose a suitable delimiter? The delimiter must be standard if we're going to have apps develop for it, and it must not be used in the name of any existing file or directory. The only characters that aren't used in filenames today are '/' and '\0'. I had a long essay here on how '.meta' breaks every recursive operation on directories (rm -rf, tar, mkisofs), but I deleted it when I remembered that I played around with the alpha/beta reiser4 which had a 'metas' dir that did the same thing, but was hidden from the normal directory listing. 'cd metas' worked, 'ls metas' worked, but 'ls' showed no directory called 'metas'. And if I try to create a file or directory called metas? I don't *think* there are any apps that will break if you tell them to open a path that doesn't exist in a directory listing. That is, typing 'vim /home/metas/acl' should always let me edit the acl on /home, and vim should never complain about how /home/metas doesn't show up in /home. A program would have to be pretty retarded to fail on something like that. Think a GUI that reads the directory to find out what is available and present them as icons for mousing around. Then there is no way to futz around with your metadata. Think filename completion, bash style. No, not just bash uses this. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
On Mon, Jul 11, 2005 at 10:33:24PM -0400, Horst von Brand wrote: Hans Reiser [EMAIL PROTECTED] wrote: I chose '' (four dots) because it clashes with less, not three dots. Is this some kind of My dots are more than yours contest?! /None/ of them is safe. .meta, ..., , .this.has.five.dots. are all perfectly legal file (or directory) names, POSIXly. If any one of them won't do, none will. Correct. I suggest lost+found. It's also a legal name but was always somewhat special. [not sure if I should place a smiley here] -- ciao - Stefan
Re: reiser4 plugins
Horst von Brand wrote: Hans Reiser [EMAIL PROTECTED] wrote: Stefan Smietanowski wrote: I think ... and .meta both serve as a logical delimiter. However some programs implement their own ... which would make it clash with them. Naturally if some program created a directory called .meta we're equally screwed. I chose '' (four dots) because it clashes with less, not three dots. Is this some kind of My dots are more than yours contest?! /None/ of them is safe. .meta, ..., , .this.has.five.dots. are all perfectly legal file (or directory) names, POSIXly. If any one of them won't do, none will. There is a long history of encroaching namespaces by creating new keywords in CS, and it is a survivable problem. Clearcase (the version control filesystem) is an excellent example. They have special meanings for @ and some other common characters, and you have to learn how to escape those characters if you use them in Clearcase. It works, it makes users happy, in practice it is far less of a problem than one might think. I think there were two or three times I had to remember how to escape things in 2 years of using it. Better to spend one's mind looking for bugs instead of this issue.
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hubert Chan wrote: On Thu, 30 Jun 2005 15:52:25 -0400, Horst von Brand [EMAIL PROTECTED] said: This doesn't even invalidate the userland VFSs of the other guys, they're still needed for systems whose kernels don't have a metadata facility. So the metadata facility in kernel won't be used, for portability's sake. Oh gee. Every operating system does threads differently. Mozilla has an abstraction layer called nspr that allows them to handle threads portably. glib/gtk has their own threads abstraction. On Windows, nspr will use the Windows method for handling threads. On Linux, it will use the Linux way. On systems that don't support threads, it can usually emulate it using timers. It's the exact same thing with the userspace VFS. If GNOME needs to handle extended attributes, it can use one mechanism under one operating system, or emulate it using some ugly hack on operating systems that don't support extended attributes. Isn't that the whole point of having a VFS? So basically if I write a program that works in both Gnome and KDE I should (according to your description) implement my own VFS that will use the Gnome or KDE VFS that will then use the OS VFS. Is it only me finding that a little silly? I mean, if I am to have the same functionality under neither Gnome nor VFS and they don't support something I need I _NEED_ a vfs so that my program is so totally independent on anything at all. My program calling My VFS which calls KDE/Gnome's VFS which calls the OS VFS will be slowe than just calling the VFS immidiately - I do hope you can see that. // Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (MingW32) iD8DBQFC0GUxBrn2kJu9P78RApViAJ4q6BrVh0H19S4pN+Zc0bqh7zk2sgCeLRVK 8b+qlg2BHjwxg8HnVANQ5XA= =2uNQ -END PGP SIGNATURE-
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Why not implement it inside the directory containg the file ? Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo This should be suit both camps I'd think? I mean, editing something is easy and you don't have to know how to navigate /meta and you don't have the clash of files vs metadata (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of the dir stesmi ?). /home/stesmi/foo - dir /home/stesmi/.meta/foo - dir containing all metadata /home/stesmi/.meta/foo/attrib - some attribute called attrib /home/stesmi/foo/bar - file /home/stesmi/foo/.meta/bar - dir containg all metadata /home/stesmi/foo/.meta/bar/attrib - some attribute called attrib The file is $dir/$file. The attrib dir is $dir/.meta/$file. The attribute is $dir/.meta/$file/$attribute. If $attrib is something user-editable it's easy to $EDITOR $dir/.meta/$file/$attrib If this has already been taken up, I must've missed it. // Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (MingW32) iD8DBQFC0fBsBrn2kJu9P78RAlt4AJ4qWik6hA4oXBNZMdZ1TkweYrJHmgCdFAY+ m+Qtg9uqBq9m0qKfRkK6iUI= =ghWb -END PGP SIGNATURE-
Re: reiser4 plugins
Hubert Chan wrote: On Wed, 06 Jul 2005 23:42:50 -0700, Hans Reiser [EMAIL PROTECTED] said: Oh no, don't store the whole path, store just the parent list. This will make fsck more robust in the event that the directory gets clobbered by hardware error. I don't think it affects the cost of detecting cycles though, I think it only makes fsck more robust. It doesn't affect the worst-case cost of detecting cycles; by necessity, any (static [1]) directed cycle detection algorithm must take O(n) time, n being the size of the tree (nodes + links). However, under reasonable assumptions (where the reasonableness of those assumptions may be questioned, but I think they're reasonable), I think that doing a depth-first-search through the parent links makes the algorithm not-too-terrible. Namely, the reasonable assumptions are that a node doesn't have too many parents, and the filesystem hierarchy is not too deep. And, remember, there are other things affected by depth of the tree anyway. For that matter, most of the time, when you push a system past reasonable assumptions, you hit performance issues, if not stability ones. Most everything but Reiser assumes that you won't have too many files in a directory, or too many small files in the FS as a whole. I think it's fair to assume people will keep things reasonable, especially if we tell them what reasonable means. As in, This is the kind of organization which Reiser4 handles really well, and this is the kind of organization which will absolutely kill your performance.
Re: reiser4 plugins
Doug Wicks wrote: How do I get off the mail list here? [EMAIL PROTECTED] See www.namesys.com, click on Join Mail List then in Unsubscribe Mailinglist and follow instructions. Very difficult, i know.
Re: reiser4 plugins
Horst von Brand wrote: Hans Reiser [EMAIL PROTECTED] wrote: [...] I think the exokernel approach by Frans is a very interesting approach. I wish I had the experience with it necessary to know if it was effective. I do NOT take the position that name resolution should be in the kernel. I DO take the position that it should be either in the kernel or out of the kernel, and should constitute one cohesive and coherent body of code. Right. If someone talks Linus into trying the exokernel approach, Are you nuts?! Such radical experiments do /not/ belong in the kernel on which millions of machines depend! Your response is a flame, and I will not respond to it.
Re: reiser4 plugins
Jonathan Briggs wrote: On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote: Hubert Chan wrote: And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too Hey, sounds like the idea I proposed a couple months ago of storing the path names in each file, instead of in directories. Only better, since each path component isn't text but a link instead. It still has the performance and locking problem of having to update every child file when moving a directory tree to a new parent. On the other hand, maybe the benefit is worth the cost. Oh no, don't store the whole path, store just the parent list. This will make fsck more robust in the event that the directory gets clobbered by hardware error. I don't think it affects the cost of detecting cycles though, I think it only makes fsck more robust. Hans
Re: reiser4 plugins
On Fri, Jul 01, 2005 at 05:54:46PM +0200, David Weinehall wrote: Which would neither need VFS changes nor be dependent on Reiser4 in any way, so I don't see why this thread lives on. Just get down to business and implement this metafs =) I've been gone for a while and suddenly drowning in mail... Anyway, I don't really like the metafs thing. To access the data, you still need to refactor userspace, so that's not a real advantage. Doing lookups from /meta all the time, instead of in the file-as-dir-whatever... And the best thing to do would be to bring these Reiser4-specific enhancements to every FS. -- mjt signature.asc Description: Digital signature
Re: reiser4 plugins
Markus Törnqvist wrote: On Fri, Jul 01, 2005 at 05:54:46PM +0200, David Weinehall wrote: Which would neither need VFS changes nor be dependent on Reiser4 in any way, so I don't see why this thread lives on. Just get down to business and implement this metafs =) I've been gone for a while and suddenly drowning in mail... Anyway, I don't really like the metafs thing. To access the data, you still need to refactor userspace, so that's not a real advantage. Doing lookups from /meta all the time, instead of in the file-as-dir-whatever... I don't really see the disadvantage. Also, metafs means much less of a fight to get people to adopt the whole meta concept, because it can be done in a POSIX-compliant way which doesn't break tar. File-as-dir is nice if you're using meta files, but it causes lots of unexpected weirdness. I don't think metafs costs us much in performance, and with one or two shell scripts, it wouldn't cost us that much efficiency on the commandline. But, I also like file-as-dir. I think it might be time for a vote. I vote metafs. Or, maybe Hans needs to tell us which way we're going... And the best thing to do would be to bring these Reiser4-specific enhancements to every FS. Which has nothing to do with whether it's done in metafs or not.
Re: reiser4 plugins
Hubert Chan wrote: On Tue, 05 Jul 2005 20:50:08 -0400 EDT, Alexander G. M. Smith [EMAIL PROTECTED] said: That sounds equivalent to no hard links (other than the usual parent directory one). If there's any directory with two links to it, then there will be a cycle somewhere! What we want is no directed cycles. That is A is the parent of B is the parent of C is the parent of A. We don't care about A is the parent of B is the parent of C; A is the parent of B is the parent of C. OK, here's a random idea that just popped into my head, and to which I've given little thought (read: none whatsoever), and may be the stupidest idea ever proposed on LKML, but thought I would just toss it out to see if it could stimulate someone to come up with something better (read: sane): Conceptually, foo/ is just a symlink to /meta/[filesystem]/[inode of foo]. Except that we want the metafiles to go away when the base file goes away. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too
Re: reiser4 plugins
hoi :) On Tue, Jul 05, 2005 at 04:32:00PM -0600, Jonathan Briggs wrote: You could do filesystems in userspace too and just use the kernel's block layer. but you can't do that in an library, you have to use a filesystem server in order to get access control. But you can build a library that handles uniform access to files and directories. Don't get me wrong, I'm all for a uniform interface for files and metadata, but I don't think that it has to be in the kernel. Gnome and KDE already have their own userspace VFS, something like that should be used. One has to distinguish between the low-level filesystem and the storage system which is presented to the user. -- Martin Waitz signature.asc Description: Digital signature
Re: reiser4 plugins
Martin Waitz wrote: hoi :) On Tue, Jul 05, 2005 at 04:32:00PM -0600, Jonathan Briggs wrote: You could do filesystems in userspace too and just use the kernel's block layer. but you can't do that in an library, you have to use a filesystem server in order to get access control. But you can build a library that handles uniform access to files and directories. Don't get me wrong, I'm all for a uniform interface for files and metadata, but I don't think that it has to be in the kernel. Gnome and KDE already have their own userspace VFS, something like that should be used. One has to distinguish between the low-level filesystem and the storage system which is presented to the user. Yes, but to do what you advocate properly, the existing semantics currently in the kernel should be moved out of it into user space. I think the exokernel approach by Frans is a very interesting approach. I wish I had the experience with it necessary to know if it was effective. I do NOT take the position that name resolution should be in the kernel. I DO take the position that it should be either in the kernel or out of the kernel, and should constitute one cohesive and coherent body of code. If someone talks Linus into trying the exokernel approach, I will be happy to educate myself to where I have an opinion on whether that works. It is easy to see powerful advantages to the exokernel approach: I wish I understood the security model for it, and I wish I was sure that name resolution would not require too many context switches as one fetches each storage component required by a name resolution. Masover's words about HURD earlier in this thread were very well said. Context switching is expensive, and I find it easier to be sure my code will both perform well and be cohesive if it is all done in the kernel by one body of code.
Re: reiser4 plugins
David Masover [EMAIL PROTECTED] wrote: [...] Just don't allow user-created hardlinks inside either metafs (/meta) or the magical meta directory inside files. And what is it useful for, after its advantage was that it was /exactly/ like regular files c, and now it is severely crippled? -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
Hubert Chan [EMAIL PROTECTED] wrote: On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey [EMAIL PROTECTED] said: Horst von Brand wrote: And who says that a normal user isn't allowed to annotate each and every file with its purpose or something else? Explain how you currently allow users to annotate arbitrary files. By keeping annotations /outside/ the files. [...] The situation is even better with file-as-dir. If the administrator wants to allow users to edit the description metadata for the file foo, the administrator can set the appropriate permissions for foo/.../description, and keep foo read-only. So now root is responsible in exquisite detail for random other users being able to keep info about my files? [...] Actually, you could use something like unionfs to allow users to keep their own annotations without affecting everyone else's. Again, root has to mount that stuff for each and every user? -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
Hans Reiser wrote: Hubert Chan wrote: On Tue, 05 Jul 2005 20:50:08 -0400 EDT, Alexander G. M. Smith [EMAIL PROTECTED] said: That sounds equivalent to no hard links (other than the usual parent directory one). If there's any directory with two links to it, then there will be a cycle somewhere! What we want is no directed cycles. That is A is the parent of B is the parent of C is the parent of A. We don't care about A is the parent of B is the parent of C; A is the parent of B is the parent of C. OK, here's a random idea that just popped into my head, and to which I've given little thought (read: none whatsoever), and may be the stupidest idea ever proposed on LKML, but thought I would just toss it out to see if it could stimulate someone to come up with something better (read: sane): Conceptually, foo/ is just a symlink to /meta/[filesystem]/[inode of foo]. Except that we want the metafiles to go away when the base file goes away. Only, /meta is a filesystem that already makes stuff go away for us, so all we have left is the issue of whether using /meta costs us performance, or whether breaking POSIX to add a symlink (such as foo/...) really gives us that much more usability. I don't know the first thing about whether it costs us performance, although it seems like it could be negligable considering the existance of mount --bind. I don't think file-as-dir gives us that much more usability, because we can always create a simple program or shell script that 'cd's us into metadata. It's still easier than having a simple program that manipulates the metadata directly, because this way we can do 'cd' and 'ls' and so on inside the metadata directory. And, once we start talking about applications, /meta will be more readily supported (as in, some apps will go through a pathname and stop when they get to a file, and then there's tar). On apps which don't have direct support for /meta, you'd be navigating to the file in question and then manually typing '...' into the dialog, so I don't see why typing '...' at the end is better than typing '/meta' or '/meta/vfs' at the beginning. That said, I'm still not entirely sure how we get /meta/vfs to work other than adding a '...' sort of delimiter anyway. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too Doesn't it make directory operations slower, too? And, will it require a format change?
Re: reiser4 plugins
Horst von Brand wrote: David Masover [EMAIL PROTECTED] wrote: [...] Just don't allow user-created hardlinks inside either metafs (/meta) or the magical meta directory inside files. And what is it useful for, after its advantage was that it was /exactly/ like regular files c, and now it is severely crippled? The advantage was never that meta files look exactly like regular files, but that they look enough like regular files that you can use existing tools on them. Of course, there are some times when you want meta files to look exactly like regular files, as in (I hate to bring it up again, but...) zip files. So, a zip file (/meta/vfs/some/zip/file/.../contents/) would allow hardlinks between files inside the zipfile, but not outside of it. This would be like creating a separate mountpoint for the zipfile's contents, and doing mount --bind when a hardlink to the zipfile is created. I'm not sure if we actually have to pretend it's a mountpoint, or if we can just take the checking that mountpoints already do and generalize it. Can you think of a way in which hardlinks are useful in /meta, in a *general* way, instead of within a specific directory controlled by a specific plugin?
Re: reiser4 plugins
Horst von Brand wrote: Hubert Chan [EMAIL PROTECTED] wrote: On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey [EMAIL PROTECTED] said: Horst von Brand wrote: And who says that a normal user isn't allowed to annotate each and every file with its purpose or something else? Explain how you currently allow users to annotate arbitrary files. By keeping annotations /outside/ the files. [...] The situation is even better with file-as-dir. If the administrator wants to allow users to edit the description metadata for the file foo, the administrator can set the appropriate permissions for foo/.../description, and keep foo read-only. So now root is responsible in exquisite detail for random other users being able to keep info about my files? If it's the general info that's associated with the file, and may even be stored inside the file, then yes, that's fair. Although I could certainly imagine foo/.../descriptions being a directory that's world-writable, allowing each user to maintain their own file inside of it. You can even set these per-user descriptions to be stored somewhere else, like the user's home directory, and that could work for CDs. Actually, you could use something like unionfs to allow users to keep their own annotations without affecting everyone else's. Again, root has to mount that stuff for each and every user? Why is that a problem? Put it in a script. Mount each user's unionfs at boot. And it's something like unionfs -- maybe it's a feature of metafs or reiserfs that we haven't thought of yet. It certainly can't be unionfs as it stands, as unionfs doesn't work on top of any reiser.
Re: reiser4 plugins
so all we have left is the issue of whether using /meta costs us performance, or whether breaking POSIX to add a symlink (such as foo/...) really gives us that much more usability. IMHO '/meta' isn't such a good idea, because a chrooted application won't be able to use it.
Re: reiser4 plugins
Hans Reiser wrote: If we also add to this the restriction that once you create the file part of a file-directory, you can never hardlink to a child of it, it should then all work, yes? So, /filename//owner should be able to avoid colliding with any common names by virtue of the '', and not letting any filedir (file-directory) have children with links to them should also work. The one thing that seems inelegant is that when you create the file part of a filedir, you must check all its children for hardlinks and fail if they exist, and you must flag all its directory children so that the plugins for them will disallow hardlinks to their children from that point onward. Still, seems workable Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Basically, we are entirely POSIX compliant outside of metafs, so that /filename is a file and may be hardlinked, and /directory/ is a directory and may not. No problems yet. Inside /meta/inode, we have no problems yet because any hardlinks created outside /meta won't even show up as hardlinks here, so we don't get hardlinked directories. Inside /meta/vfs, where we can find the metadata of '/filename' under '/meta/vfs/filename/...', we have what looks like a problem -- we may have hardlinks created outside metafs, which will show up as two directories inside metafs, so those directories are hardlinks. I don't think that's such a problem, since we won't allow users to change anything in /meta/vfs except when it's inside metadata, such as '/meta/vfs/some/where/...'. Thus, it's impossible to create a hardlink to a directory unless we do it *outside* metafs, as a hardlink to a file. Now, the question I asked is, what if we want hardlinks *inside* metafs, *inside* the metadata for a given file, and would we ever want such a thing? Because we can avoid the whole problem if we just disallow any sys_link calls inside metafs. Of course, sometimes we want to have a chunk of metadata that appears *exactly* as if it were a normal, POSIX-compliant directory tree, such as the contents of a tarball or a zipfile. In that case, we might want to have the zip plugin allow hardlinks inside /meta/file.zip/.../contents -- the only restriction is that any hardlink made to a file inside 'contents' must be made to another file inside 'contents'. Hans Reiser wrote: David Masover wrote: Now, can anyone think of a situation where we want user-created hardlinks inside metadata? More importantly, what do we do about it? I think the equivalent of symlinks would be good enough to get by on for now for most linking of metafiles. Maybe some years from now somebody can fault me for saying this and write a patch to fix it to be better, at which point I will be happy to concede the point. So the basic principal here is, one can have hardlinks to directories without cycles provided that one does not allow any child of the directory to have a hardlink. The question is, how cleanly can that relaxed restriction be coded? Hans
Re: reiser4 plugins
Adrian Ulrich wrote: so all we have left is the issue of whether using /meta costs us performance, or whether breaking POSIX to add a symlink (such as foo/...) really gives us that much more usability. IMHO '/meta' isn't such a good idea, because a chrooted application won't be able to use it. mount --bind. Is there a performance hit for having too many of those? mount --bind /meta/vfs/some/chroot /some/chroot/meta Also, maybe a separately-mounted /meta, maybe with something like '-o root=' I can't think of when you'd have a chrooted application that uses /meta, but wasn't written with /meta in mind, so as to have these mount commands in its init scripts.
Re: reiser4 plugins
mount --bind /meta/vfs/some/chroot /some/chroot/meta This maybe funny if you got 1-2 chrooted applications. But it will be a nightmare if you got 20-30 chrooted applications. -- We're working on it, slowly but surely...or not-so-surely in the spots we're not so sure... -- Larry Wall
Re: reiser4 plugins
Hans Reiser [EMAIL PROTECTED] wrote: [...] I think the exokernel approach by Frans is a very interesting approach. I wish I had the experience with it necessary to know if it was effective. I do NOT take the position that name resolution should be in the kernel. I DO take the position that it should be either in the kernel or out of the kernel, and should constitute one cohesive and coherent body of code. Right. If someone talks Linus into trying the exokernel approach, Are you nuts?! Such radical experiments do /not/ belong in the kernel on which millions of machines depend! Go and fork off a branch to play around with this, and if it does show real promise, you can then come back and try to integrate this into the official kernel. I will be happy to educate myself to where I have an opinion on whether that works. It is easy to see powerful advantages to the exokernel approach: I wish I understood the security model for it, and I wish I was sure that name resolution would not require too many context switches as one fetches each storage component required by a name resolution. Exactly the kinds of questions that have to get solid answers before any experimental patches can get off the ground. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
David Masover wrote: And, once we start talking about applications, /meta will be more readily supported (as in, some apps will go through a pathname and stop when they get to a file, and then there's tar). On apps which don't have direct support for /meta, you'd be navigating to the file in question and then manually typing '...' into the dialog, so I don't see why typing '...' at the end is better than typing '/meta' or '/meta/vfs' at the beginning. Performance. If you type it at the end, and you already have done the lookup of the filename, then you can go from the file to one of its methods, instead of a complete new traversal of another tree under /meta That said, I'm still not entirely sure how we get /meta/vfs to work other than adding a '...' sort of delimiter anyway. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too Doesn't it make directory operations slower, too? Not sure. It consumes space though. And, will it require a format change? Yes, but we have plugins now, so.
Re: reiser4 plugins
Nate Diller wrote: as an example, if a program were to store some things it needs access to in its executable's attributes, it should have the option of keeping a hard reference to something, so that it can't be deleted out from underneath. this enables sane sharing of resources without ownership tracking problems (see windows DLL hell for details). the attribute space should be indistinguishable from the rest of the namespace, and should be able to link (soft or hard) anywhere in the FS. anything less is too much work for too little reward. You already have a problem with hardlinks not crossing mount points, but I understand your point. If we can write code for solving the cycle problem cleanly, it would be best.
Re: reiser4 plugins
On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote: Hubert Chan wrote: And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too Hey, sounds like the idea I proposed a couple months ago of storing the path names in each file, instead of in directories. Only better, since each path component isn't text but a link instead. It still has the performance and locking problem of having to update every child file when moving a directory tree to a new parent. On the other hand, maybe the benefit is worth the cost. -- Jonathan Briggs [EMAIL PROTECTED] eSoft, Inc. signature.asc Description: This is a digitally signed message part
Re: reiser4 plugins
On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs [EMAIL PROTECTED] said: On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote: Hubert Chan wrote: And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too If you can store the parents, then finding cycles (relatively) quickly is pretty easy: before you try to make A the parent of B, walk up the parent pointers starting from A. If you ever reach B, you have a cycle. If not, you don't have a cycle. (Hmmm. Do I need a proof of correctness for this? It's just depth-first-search, which everybody learned in their first algorithms course.) Running time is (at most) just the sum of all (directed) path lengths from A to the root. Assuming people don't have too deep of a hierarchy, or too much branching with hard links (i.e. nodes with too many parents), which I think is a reasonable assumption, running time should be fairly quick. (People may have millions of files in a single directory, which is why doing the DFS in the forward direction is a bad idea. But people probably don't have millions of hard links to the same file, or hierarchies that are millions of levels deep.) And it only happens when a hard link is created -- probably not a very performance-dependent event (i.e. it's not the scheduling algorithm). I would imagine that some creative caching could be done to speed up the situation where you try to make a whole batch of hard links. (When you walk up the parent pointers starting from A, cache the inode numbers that you encounter. If you then try to make A the parent of C, check if C is one of the inode numbers in the cache. If yes, we have a cycle; if no, we don't have a cycle.) Extra disk storage is just about one extra word for most nodes (that only have one parent -- basically, one extra word per parent), if you can pack the data efficiently. (i.e. we don't want to waste a extra whole block per node.) Of course, since this requires extra storage on the part of the filesystem, the (kernel) VFS change would have to be something where the VFS asks the filesystem whether making A the parent of B will create a cycle; it's not something the VFS can do transparently for everybody. Filesystems that don't store this extra stuff just return yes if B is a directory (thus disallowing the link), and no if B is a file, and we get the same situation we have right now. It would require a disk format change (sort of) in Reiser4, but (as Hans mentioned) since we have plugins, shouldn't require a reformat (as I understand it). Just run a tool that builds the parent-pointer data; until you run that tool, the filesystem can just disallow hard links. After you run the tool, updating parent pointers is handled automatically by the filesystem. Hey, sounds like the idea I proposed a couple months ago of storing the path names in each file, instead of in directories. Only better, since each path component isn't text but a link instead. It still has the performance and locking problem of having to update every child file when moving a directory tree to a new parent. On the other hand, maybe the benefit is worth the cost. Every node should store the inode number(s) for its parent(s). Not the path name. So you don't need to do any updates, since when you move a tree, inode numbers don't change. -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
On Tue, 05 Jul 2005 22:51:07 -0400, Horst von Brand [EMAIL PROTECTED] said: Hubert Chan [EMAIL PROTECTED] wrote: On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey [EMAIL PROTECTED] said: Horst von Brand wrote: And who says that a normal user isn't allowed to annotate each and every file with its purpose or something else? Explain how you currently allow users to annotate arbitrary files. By keeping annotations /outside/ the files. So if I want to share annotations, I have to look in 20 different places? [...] The situation is even better with file-as-dir. If the administrator wants to allow users to edit the description metadata for the file foo, the administrator can set the appropriate permissions for foo/.../description, and keep foo read-only. So now root is responsible in exquisite detail for random other users being able to keep info about my files? I can grant people permissions to write random info into my own files. Or they can use unionfs if I don't grant them permissions. Remember: the above argument was citing an advantage of file-as-dir over packed files (storing metadata as a tar or zip file, similar to what OpenOffice.org does, or even something like exif or id3). In a packed file, I can't allow people to edit the description attribute without allowing them to edit the entire file. With file-as-dir, I get much finer grained permissions. [...] Actually, you could use something like unionfs to allow users to keep their own annotations without affecting everyone else's. Again, root has to mount that stuff for each and every user? suid program that allows union mount into a directory within my own tree (or just into any directory that I have write permissions should be sufficiently secure). -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
On Wed, 2005-07-06 at 15:51 -0400, Hubert Chan wrote: On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs [EMAIL PROTECTED] said: [snip] It still has the performance and locking problem of having to update every child file when moving a directory tree to a new parent. On the other hand, maybe the benefit is worth the cost. Every node should store the inode number(s) for its parent(s). Not the path name. So you don't need to do any updates, since when you move a tree, inode numbers don't change. You do need the updates if you change what inode is the parent. /a/b/c, /a/b/d, /a/b/e, /b mv /a/b /b Now you have to change the stored grand-parent inodes for /a/b/c, /a/b/d and /a/b/e so that they reference /b/b instead of /a/b. -- Jonathan Briggs [EMAIL PROTECTED] eSoft, Inc. signature.asc Description: This is a digitally signed message part
Re: reiser4 plugins
Hubert Chan [EMAIL PROTECTED] wrote: On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs [EMAIL PROTECTED] said: On Tue, 2005-07-05 at 23:44 -0700, Hans Reiser wrote: Hubert Chan wrote: And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too If you can store the parents, then finding cycles (relatively) quickly is pretty easy: before you try to make A the parent of B, walk up the parent pointers starting from A. If you ever reach B, you have a cycle. If not, you don't have a cycle. (Hmmm. Do I need a proof of correctness for this? It's just depth-first-search, which everybody learned in their first algorithms course.) Correct. And you need space for potentially a huge lot of up pointers for each file. And then there is the (very minor) problem is that meanwhile /nothing/ can touch the filesystem to do any change... How do you delete such an object? You will have to delete from each child down to the first object that has a pointer to it from the outside, if you don't want garbage left behind. And that means walking down to /each/ reachable object, then walking up from there to see if all its parents are in the DAG rooted at what you are going to delete. This means potentially walking through the whole filesystem (if you want to delete the root, or something near). You will run out of memory, and again, meanwhile no changes can be allowed. It is for a reason that Unix filesystems don't have up pointers for files, and just leaves (i.e., files) can have multiple hard links. And this /was/ explained in detail the last flamefest over this exact same point. I thought the ReiserFS people had learned from that... -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
Hans Reiser wrote: David Masover wrote: And, once we start talking about applications, /meta will be more readily supported (as in, some apps will go through a pathname and stop when they get to a file, and then there's tar). On apps which don't have direct support for /meta, you'd be navigating to the file in question and then manually typing '...' into the dialog, so I don't see why typing '...' at the end is better than typing '/meta' or '/meta/vfs' at the beginning. Performance. If you type it at the end, and you already have done the lookup of the filename, then you can go from the file to one of its methods, instead of a complete new traversal of another tree under /meta Only, it's a different view of the same tree. That may not matter performance-wise, though. Also, for performance-critical applications, the /meta tree is pretty easy -- it becomes more like /meta/inode/some_inode_number/. There's also the chroot issue, though. That said, I'm still not entirely sure how we get /meta/vfs to work other than adding a '...' sort of delimiter anyway. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too Doesn't it make directory operations slower, too? Not sure. It consumes space though. And, will it require a format change? Yes, but we have plugins now, so. So, will the format change happen at mount time? Will it need a special mount flag? Will I need to use debugfs or some other offline tool?
Re: reiser4 plugins
Jonathan Briggs wrote: On Wed, 2005-07-06 at 15:51 -0400, Hubert Chan wrote: On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs [EMAIL PROTECTED] said: [snip] It still has the performance and locking problem of having to update every child file when moving a directory tree to a new parent. On the other hand, maybe the benefit is worth the cost. Every node should store the inode number(s) for its parent(s). Not the path name. So you don't need to do any updates, since when you move a tree, inode numbers don't change. You do need the updates if you change what inode is the parent. /a/b/c, /a/b/d, /a/b/e, /b mv /a/b /b Now you have to change the stored grand-parent inodes for /a/b/c, /a/b/d and /a/b/e so that they reference /b/b instead of /a/b. no, c, d, and e all would reference /a/b's *inode*, not *pathname*. Then /a/b would reference /a. So with the above mv, you just change the reference in /a/b (now /b/b) to point to /b as parent. Only way you would have to touch c, d, and e is if you did: mkdir /b/b mv /a/b/* /b/b/ but, since you're also essentially unlinking them from /a/b and linking them into /b, it's not that much more of a performance hit to update one pointer in the file.
Re: reiser4 plugins
On Wed, 06 Jul 2005 16:33:23 -0400, Horst von Brand [EMAIL PROTECTED] said: Hubert Chan [EMAIL PROTECTED] wrote: If you can store the parents, then finding cycles (relatively) quickly is pretty easy: before you try to make A the parent of B, walk up the parent pointers starting from A. If you ever reach B, you have a cycle. If not, you don't have a cycle. (Hmmm. Do I need a proof of correctness for this? It's just depth-first-search, which everybody learned in their first algorithms course.) Correct. And you need space for potentially a huge lot of up pointers for each file. People (that I know of) don't normally have a huge lot of hardlinks to a single file. And then there is the (very minor) problem is that meanwhile /nothing/ can touch the filesystem to do any change... If the DFS is quick, it shouldn't make much difference. Lock nodes as you walk up the tree, and people can change unlocked nodes without harming anything. All you need to do is make sure that no up pointers get added to nodes that you've already looked at. People can also still edit files without changing the tree (and hence not screwing up the DFS). How do you delete such an object? You will have to delete from each child down to the first object that has a pointer to it from the outside, if you don't want garbage left behind. And that means walking down to /each/ reachable object, then walking up from there to see if all its parents are in the DAG rooted at what you are going to delete. This means potentially walking through the whole filesystem (if you want to delete the root, or something near). You will run out of memory, and again, meanwhile no changes can be allowed. What does deleting have to do with up pointers? If you delete a directory that has refcount greater than 1, you decrease the refcount, remove the appropriate up pointer, and remove the inode entry from its parent. If you delete a directory that has refcount equal to 1, well, we're in the same situation we're in right now when you delete a directory; you need to rm -r, which doesn't lock up the whole filesystem. -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
RE: reiser4 plugins
Can you please take me off your email list? I have sent numerous emails and still nobody will assist me. Please, can you help me with this or should I contact your provider? [EMAIL PROTECTED] Doug -Original Message- From: Hans Reiser [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 05, 2005 11:45 PM To: Hubert Chan Cc: Alexander G. M. Smith; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; linux-kernel@vger.kernel.org; reiserfs-list@namesys.com; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: reiser4 plugins Hubert Chan wrote: On Tue, 05 Jul 2005 20:50:08 -0400 EDT, Alexander G. M. Smith [EMAIL PROTECTED] said: That sounds equivalent to no hard links (other than the usual parent directory one). If there's any directory with two links to it, then there will be a cycle somewhere! What we want is no directed cycles. That is A is the parent of B is the parent of C is the parent of A. We don't care about A is the parent of B is the parent of C; A is the parent of B is the parent of C. OK, here's a random idea that just popped into my head, and to which I've given little thought (read: none whatsoever), and may be the stupidest idea ever proposed on LKML, but thought I would just toss it out to see if it could stimulate someone to come up with something better (read: sane): Conceptually, foo/ is just a symlink to /meta/[filesystem]/[inode of foo]. Except that we want the metafiles to go away when the base file goes away. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too
RE: reiser4 plugins
How do I get off the mail list here? [EMAIL PROTECTED] -Original Message- From: Hans Reiser [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 06, 2005 3:50 PM To: David Masover Cc: Hubert Chan; Alexander G. M. Smith; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; linux-kernel@vger.kernel.org; reiserfs-list@namesys.com; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: reiser4 plugins David Masover wrote: So, will the format change happen at mount time? Will it need a special mount flag? Will I need to use debugfs or some other offline tool? First we make sure we have the right answer. Have we solved the cycle problem? Can we run out of memory as Horst/Nikita suggest?
Re: reiser4 plugins
Hubert Chan wrote: On Wed, 06 Jul 2005 16:33:23 -0400, Horst von Brand [EMAIL PROTECTED] said: Hubert Chan [EMAIL PROTECTED] wrote: If you can store the parents, then finding cycles (relatively) quickly is pretty easy: before you try to make A the parent of B, walk up the parent pointers starting from A. If you ever reach B, you have a cycle. If not, you don't have a cycle. (Hmmm. Do I need a proof of correctness for this? It's just depth-first-search, which everybody learned in their first algorithms course.) Correct. And you need space for potentially a huge lot of up pointers for each file. People (that I know of) don't normally have a huge lot of hardlinks to a single file. And speaking of which, the only doomsday scenario (running out of RAM) that I can think of with this scheme is if we have a ton of hardlinks to the same file and we try to move one of them. But this scales linearly with the number of hardlinks, I think. Maybe not quite, but certainly not exponentially. The only other doomsday scenario is if we have a ludicrously deep tree. To make this work in real usage, not DOS testing, we really need both of those, and even then I'm not sure it can work. What's the maximum number of hardlinks supported to a single file? What's the maximum tree depth? Can these be limited to prevent actual DOS attempts? Now, if we were to ever actually *create* a cycle, then yes, we might end up traversing the whole tree to delete it, possibly running out of RAM. But we don't ever actually create cycles except if we were to have corruption, which could as easily create cycles in any other FS.
Re: reiser4 plugins
On 07/06/05 03:54:09PM -0700, Doug Wicks wrote: How do I get off the mail list here? Read the auto-appended signature at the bottom of every message. [EMAIL PROTECTED] Jim. -Original Message- From: Hans Reiser [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 06, 2005 3:50 PM To: David Masover Cc: Hubert Chan; Alexander G. M. Smith; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; linux-kernel@vger.kernel.org; reiserfs-list@namesys.com; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: reiser4 plugins David Masover wrote: So, will the format change happen at mount time? Will it need a special mount flag? Will I need to use debugfs or some other offline tool? First we make sure we have the right answer. Have we solved the cycle problem? Can we run out of memory as Horst/Nikita suggest? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
On Wed, Jul 06, 2005 at 09:33:13PM -0500, David Masover wrote: And speaking of which, the only doomsday scenario (running out of RAM) that I can think of with this scheme is if we have a ton of hardlinks to the same file and we try to move one of them. But this scales linearly with the number of hardlinks, I think. Maybe not quite, but certainly not exponentially. The only other doomsday scenario is if we have a ludicrously deep tree. rename(a/b, c/b), if a and c are identical. End result, either you deadlock trying to lock the same object twice, or you end up removing b since the target of a rename is unlinked. The VFS uses dentries, there is one per hardlinked object, and they have a single parent only. So a/b and c/b are represented by the same inode, but they have completely different dentry-aliases associated with them. Similar things for removal, we can safely remove a file, but not a directory that still has children, if you have 'meta-files' hanging of a file, you'd have to get rid of the metadata objects first. If you want to use a file as a directory, it probably will need the same restrictions as a directory if you expect the dcache and VFS locking to work correctly. So that means, _no hardlinks to files_ (the file system could internally implement copy-on-write type links, or use a content addressable storage to deal with diskspace issues) and the file system probably has to d_unhash/destroy metadata objects before it can unlink the file object, etc. To make this work in real usage, not DOS testing, we really need both of those, and even then I'm not sure it can work. What's the maximum number of hardlinks supported to a single file? I believe nlink is a 16-bit value, so that would be either 32K or 64K depending on signedness. What's the maximum tree depth? Last time I checked, PATH_MAX is 4096, so that would be about 2048 single character directory names. Can these be limited to prevent actual DOS attempts? Is that the 'nobody needs more than 64KB of memory' kind of DOS? Jan
Re: reiser4 plugins
hoi :) On Wed, Jun 29, 2005 at 08:04:58PM -0700, Hans Reiser wrote: How is directories as files logically any different than putting all data into .data files and making all files directories (yes you would need some sort of special handling for files that were really called .data). Add to this that you make .data the default if the file within the directory is not specified, and define a stanadard set of names for metafiles, and you've got the essential idea, and any differences are details. sure, that would work more or less, but what would it give you? You haven't introduced anything new that userspace couldn't do before. Just write a library which mangles pathnames and treats files and directories the same. You don't need the kernel to do that. Filesystems are there to store files. Everything else can be done in userspace. -- Martin Waitz signature.asc Description: Digital signature
Re: reiser4 plugins
On Thu, 30 Jun 2005 15:52:25 -0400, Horst von Brand [EMAIL PROTECTED] said: This doesn't even invalidate the userland VFSs of the other guys, they're still needed for systems whose kernels don't have a metadata facility. So the metadata facility in kernel won't be used, for portability's sake. Oh gee. Every operating system does threads differently. Mozilla has an abstraction layer called nspr that allows them to handle threads portably. glib/gtk has their own threads abstraction. On Windows, nspr will use the Windows method for handling threads. On Linux, it will use the Linux way. On systems that don't support threads, it can usually emulate it using timers. It's the exact same thing with the userspace VFS. If GNOME needs to handle extended attributes, it can use one mechanism under one operating system, or emulate it using some ugly hack on operating systems that don't support extended attributes. Isn't that the whole point of having a VFS? -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey [EMAIL PROTECTED] said: Horst von Brand wrote: And who says that a normal user isn't allowed to annotate each and every file with its purpose or something else? Explain how you currently allow users to annotate arbitrary files. I can very well imagine a system in which users (say students in a Linux class) want to do so... on a shared machine. Or users having a shared MP3 or photograph or ... collection, with individual notes on each. Or even developers wanting to annotate source code files with their comments, but leave them read-only (or have them under SCM). This same argument could be used to attack the idea of group permissions -- that groups of users might have conflicting goals. Implementing metadata in userspace via bundled files has the same drawback. The situation is even better with file-as-dir. If the administrator wants to allow users to edit the description metadata for the file foo, the administrator can set the appropriate permissions for foo/.../description, and keep foo read-only. Kevin Bowen [EMAIL PROTECTED] wrote: If you're sysadmining a multiuser reiser4 box, and your users are able to modify the metadata of files they don't own, then you go to sysadmin purgatory. Actually, you could use something like unionfs to allow users to keep their own annotations without affecting everyone else's. -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
On Fri, 01 Jul 2005 03:06:19 -0500, David Masover [EMAIL PROTECTED] said: Hubert Chan wrote: The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... It was whatever was Al Viro's (technical) complaint about file-as-dir. I don't remember exactly what it was. The technical people know what it is (and the Namesys guys are probably working on it), and the exact issue doesn't concern us non-technical people that much, so I don't feel like looking it up. But if you want to, just look for Al Viro's message in this thread. -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
Hubert Chan wrote: On Fri, 01 Jul 2005 03:06:19 -0500, David Masover [EMAIL PROTECTED] said: Hubert Chan wrote: The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... It was whatever was Al Viro's (technical) complaint about file-as-dir. I don't remember exactly what it was. The technical people know what it is (and the Namesys guys are probably working on it), and the exact issue doesn't concern us non-technical people that much, so I don't feel like looking it up. But if you want to, just look for Al Viro's message in this thread. Cycle detection when hard links to directories are allowed. There is a debate over whether cycle detection is feasible that can only be resolved by working code or a formal proof that it is not computationally feasible. Hans
Re: reiser4 plugins
Hans Reiser wrote: Hubert Chan wrote: On Fri, 01 Jul 2005 03:06:19 -0500, David Masover [EMAIL PROTECTED] said: Hubert Chan wrote: The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... It was whatever was Al Viro's (technical) complaint about file-as-dir. I don't remember exactly what it was. The technical people know what it is (and the Namesys guys are probably working on it), and the exact issue doesn't concern us non-technical people that much, so I don't feel like looking it up. But if you want to, just look for Al Viro's message in this thread. Cycle detection when hard links to directories are allowed. There is a debate over whether cycle detection is feasible that can only be resolved by working code or a formal proof that it is not computationally feasible. Ah. But then, one solution was to avoid the issue at all, and have the directory inside a file act as a mountpoint. After all, mount --bind doesn't cause problems... Hey! This sounds like metafs (/meta) already! I wonder if we can do file-as-dir in /meta, and just not support user-created hardlinks there? (other than creating brand-new files, of course...)
Re: reiser4 plugins
Horst von Brand wrote: David Masover [EMAIL PROTECTED] wrote: Horst von Brand wrote: David Masover [EMAIL PROTECTED] wrote: David Weinehall wrote: On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: David Weinehall wrote: [...] Even if they don't, it would be more beneficial to me How, exactly? Go back and read. I have. And have seen /no/ benefit to you. Except, of course, the benefit accrued from some magic in Linus' kernel, by which all format differences go in a puff of smoke if they are implemented inside it, and furthermore all userland gets rebuilt to use the kernel's way overnight. Let's say cryptocompress gets implemented. Not all of userland rewritten, not even any of userland rewritten, just a cryptocompress plugin for the kernel. And instead of having to learn a new tool, I can just browse around in /meta. Yes, all of userland gets rebuilt, and the world is better for it. But *incrementally*. Not overnight. Here, the how should it be done, if at all discussion has barely started. And AFAICS there are plenty of ways of getting 95% of the advantages /without/ touching the kernel, Which advantages can we get without touching the kernel? And why does it make sense to do them outside the kernel? Because we can is a little HURD-ish... so that should be the path to follow. Nobody has shown any real evidence at all for this not being so. Go back and read again. Just because you can't get your mind around a concept doesn't mean that it's a bad concept, and that no one else can use it. I /do/ get my mind around the concept, it /is/ sometimes useful. It /can/ be done without the kernel, No, it can't. Unless you patch *all* of userland. and so /should/ be done that way. That is all. I'll refer you to my HURD comment: Just about anything done currently in the kernel can be done in userland. Witness HURD. Doesn't mean it should. Witness HURD. and probably most Linux users Most Linux users don't use experimental filesystems at all... Actually, they do -- ext3 was experimental once. Right. And ext2, and ext, and xiafs, and ... So? When they were experimental, only a handful of utter fools commited their data to them. But, they had to start somewhere. You seem hell-bent on keeping anything unproven out of the kernel and away from users. How do you suggest it prove itself, then? to have metafs supported in both GNOME and KDE, even if they still need an emulation layer to support other systems. So Gnome and KDE get larger (and thus slower) for everybody. Smaller (and thus faster) on supported systems, Sorry, but just because some $DISTRO gives ReiserFS4 as an /option/ doesn't mean they will have everything duplicated as For ReiserFS4 and For other filesystems. My assertion stands, until there are ReiserFS4-only distributions. Why not? The For Reiser4 version is just a thin layer on top of metafs. And after all, metafs can be implemented for other filesystems. And, as someone else pointed out, dealing with different systems doing things in different ways is what portability is all about. Besides, Gnome and KDE will have to agree on the formats involved first, which is /exactly/ what is supposed to be impossible unless this stuff is implemented in the kernel... I never said that, You (and others) Others, then. have told us time and again that each of them have their own incompatible ways of handling metadata, and that only if this was handled in the kernel they would make use of a common way of managing it... All I ever said was that it would be easier for them to get along, and certainly *much* easier to use existing tools (the commandline) to poke inside the metadata, whether it's compatible or not. Ultimately, maybe it leads to cooperation, maybe not. But there are immediate benefits even if it doesn't lead to cooperation. but for one thing, whether they do or not, it's nice if my shell and my editor and all the other things that I use don't have to agree on anything to manipulate the formats involved. Please, read what you just said. Everybody (kernel somewhat included) will have to agree on the format of the metadata. This is not just about GNOME/KDE. It is about GNOME/KDE not developing an additional layer, that you wouldn't like anyway, How do you know what I would or would not like? Maybe I was generalizing you. Others don't like Reiser4 as it is because it's got layering violations. GNOME/KDE developing their own VFS on top of the kernel's for no good reason is a layering violation. Am I misunderstanding? that cannot be accessed from anything except GNOME/KDE. So Gnome and KDE agree on some secret formats that only they can handle? Behind their user's backs? As open source? And happily shoot their feet
Re: reiser4 plugins
On Tue, 2005-07-05 at 17:46 +0200, Martin Waitz wrote: [snip] Filesystems are there to store files. Everything else can be done in userspace. You could do filesystems in userspace too and just use the kernel's block layer. In fact you can reduce the OS kernel to just interrupts, memory management, context switches and message passing. Everything else can be done in userspace, but that doesn't always make it a good idea. -- Jonathan Briggs [EMAIL PROTECTED] eSoft, Inc. signature.asc Description: This is a digitally signed message part
Re: reiser4 plugins
David Masover wrote: Hans Reiser wrote: Hubert Chan wrote: On Fri, 01 Jul 2005 03:06:19 -0500, David Masover [EMAIL PROTECTED] said: Hubert Chan wrote: The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... It was whatever was Al Viro's (technical) complaint about file-as-dir. I don't remember exactly what it was. The technical people know what it is (and the Namesys guys are probably working on it), and the exact issue doesn't concern us non-technical people that much, so I don't feel like looking it up. But if you want to, just look for Al Viro's message in this thread. Cycle detection when hard links to directories are allowed. There is a debate over whether cycle detection is feasible that can only be resolved by working code or a formal proof that it is not computationally feasible. Ah. But then, one solution was to avoid the issue at all, and have the directory inside a file act as a mountpoint. After all, mount --bind doesn't cause problems... Can you explain this idea at greater length? Hey! This sounds like metafs (/meta) already! I wonder if we can do file-as-dir in /meta, and just not support user-created hardlinks there? (other than creating brand-new files, of course...)
Re: reiser4 plugins
Hans Reiser wrote: David Masover wrote: Hans Reiser wrote: Hubert Chan wrote: On Fri, 01 Jul 2005 03:06:19 -0500, David Masover [EMAIL PROTECTED] said: Hubert Chan wrote: The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... It was whatever was Al Viro's (technical) complaint about file-as-dir. I don't remember exactly what it was. The technical people know what it is (and the Namesys guys are probably working on it), and the exact issue doesn't concern us non-technical people that much, so I don't feel like looking it up. But if you want to, just look for Al Viro's message in this thread. Cycle detection when hard links to directories are allowed. There is a debate over whether cycle detection is feasible that can only be resolved by working code or a formal proof that it is not computationally feasible. Ah. But then, one solution was to avoid the issue at all, and have the directory inside a file act as a mountpoint. After all, mount --bind doesn't cause problems... Can you explain this idea at greater length? Just don't allow user-created hardlinks inside either metafs (/meta) or the magical meta directory inside files. In order to do that, one way would be to have file/... appear as a mountpoint. I don't know if this is feasable, performance-wise, but I think it would work. Hey! This sounds like metafs (/meta) already! I wonder if we can do file-as-dir in /meta, and just not support user-created hardlinks there? (other than creating brand-new files, of course...) This is still my preferred solution, because it's not any harder or less efficient to develop new apps based on metafs than on file-as-dir, but it means that old apps will see something *entirely* POSIX-compliant, and the strangeness will be confined to /meta. Basically, you have to allow hardlinks in /meta, in case some plugin wants to do that, but if you have files that are also directories, you also have to make sure that users can't create hardlinks. I don't think this would be particularly hard, though. Pretend it's a read-only filesystem unless the user is doing something safe, like creating a brand-new file, deleting an old one, or modifying the contents of an existing one, all assuming that the plugin responsible for the file/directory you're doing this in allows it. But then, if we're doing metafs, I don't think you need file-as-directory, and certain existing tools that we'd like to be able to point at metafs might not like file-as-directory anyway. Now, can anyone think of a situation where we want user-created hardlinks inside metadata? More importantly, what do we do about it?
Re: reiser4 plugins
David Masover wrote: Now, can anyone think of a situation where we want user-created hardlinks inside metadata? More importantly, what do we do about it? I think the equivalent of symlinks would be good enough to get by on for now for most linking of metafiles. Maybe some years from now somebody can fault me for saying this and write a patch to fix it to be better, at which point I will be happy to concede the point. So the basic principal here is, one can have hardlinks to directories without cycles provided that one does not allow any child of the directory to have a hardlink. The question is, how cleanly can that relaxed restriction be coded? Hans
Re: reiser4 plugins
I got it slightly wrong. One can have hardlinks to a directory without cycles provided that one does not have hardlinks from the children of that directory to any file not a child of that directory. (Mountpoints currently implement that restriction.) Question: can one implement that lesser restriction above with elegant code? Is the greater restriction below easier to code? (If no to the first and yes to the second is correct, then I can accept the greater restriction described below.) One can have hardlinks to directories without cycles provided that one does not allow any child of the directory to have a hardlink. Hans
Re: reiser4 plugins
If we also add to this the restriction that once you create the file part of a file-directory, you can never hardlink to a child of it, it should then all work, yes? So, /filename//owner should be able to avoid colliding with any common names by virtue of the '', and not letting any filedir (file-directory) have children with links to them should also work. The one thing that seems inelegant is that when you create the file part of a filedir, you must check all its children for hardlinks and fail if they exist, and you must flag all its directory children so that the plugins for them will disallow hardlinks to their children from that point onward. Still, seems workable Thanks David, Hans Hans Reiser wrote: David Masover wrote: Now, can anyone think of a situation where we want user-created hardlinks inside metadata? More importantly, what do we do about it? I think the equivalent of symlinks would be good enough to get by on for now for most linking of metafiles. Maybe some years from now somebody can fault me for saying this and write a patch to fix it to be better, at which point I will be happy to concede the point. So the basic principal here is, one can have hardlinks to directories without cycles provided that one does not allow any child of the directory to have a hardlink. The question is, how cleanly can that relaxed restriction be coded? Hans
Re: reiser4 plugins
Hans Reiser wrote on Tue, 05 Jul 2005 16:56:02 -0700: One can have hardlinks to a directory without cycles provided that one does not have hardlinks from the children of that directory to any file not a child of that directory. (Mountpoints currently implement that restriction.) Question: can one implement that lesser restriction above with elegant code? Is the greater restriction below easier to code? (If no to the first and yes to the second is correct, then I can accept the greater restriction described below.) One can have hardlinks to directories without cycles provided that one does not allow any child of the directory to have a hardlink. That sounds equivalent to no hard links (other than the usual parent directory one). If there's any directory with two links to it, then there will be a cycle somewhere! The tradeoff is the size of the cycle to the amount of memory and other resources needed to clean up the cycle when rename/deletion changes the directory graph structure. Various artificial limits can be imposed, such as not creating a link which would result in a cycle of more than some number of files/directories (adjust so that cleanup fits in the memory size of the machine). Or just fail the cleanup if it is too complex (user deletes individual items in the cycle then tries again). Perhaps you are thinking about the speed improvement for cycle checking that comes from marking directories as known to not contain children pointing outside the graph rooted at that directory. Then that directory can be treated as an ordinary directory in the cycle cleanup procedure, avoiding the need to trace down the links to all its descendants. That's essentially what your mount point restriction is. - Alex
Re: reiser4 plugins
On Tue, 05 Jul 2005 20:50:08 -0400 EDT, Alexander G. M. Smith [EMAIL PROTECTED] said: That sounds equivalent to no hard links (other than the usual parent directory one). If there's any directory with two links to it, then there will be a cycle somewhere! What we want is no directed cycles. That is A is the parent of B is the parent of C is the parent of A. We don't care about A is the parent of B is the parent of C; A is the parent of B is the parent of C. OK, here's a random idea that just popped into my head, and to which I've given little thought (read: none whatsoever), and may be the stupidest idea ever proposed on LKML, but thought I would just toss it out to see if it could stimulate someone to come up with something better (read: sane): Conceptually, foo/ is just a symlink to /meta/[filesystem]/[inode of foo]. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? -- Hubert Chan [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: reiser4 plugins
On Tuesday July 5, [EMAIL PROTECTED] wrote: I got it slightly wrong. One can have hardlinks to a directory without cycles provided that one does not have hardlinks from the children of that directory to any file not a child of that directory. (Mountpoints currently implement that restriction.) Question: can one implement that lesser restriction above with elegant code? Is the greater restriction below easier to code? (If no to the first and yes to the second is correct, then I can accept the greater restriction described below.) technical-content I think the lesser restriction above can be implemented elegantly, but it would require major dcache surgery. Currently all the dentries of names in a directory are linked to the dentry of the directory. As you would have to let a directory have multiple dentries, it would be best to change that linkage so that the dentries of names in a directory were linked to the inode of that directory (of which there is still only one). Thus instead of just having a dentry tree with inodes attached at each point, you would have a dentry/inode tree with inodes a more integral part of the tree. (i.e. the path from the root down would be dentry - inode - dentry -inode - dentry etc). This would have major implications for the current code. The greater restriction below should be easy to code providing you were willing to have two sorts of directories: those which could be linked (ie. they sometimes look like files) and those which cannot (they never look like files). Then for each dentry, you remember the closest parent which is a can-be-linked directory an make sure a hard-link will never want to change the can-be-linked directory for the target. If you want to be more general, and have only one sort of directory that just behaves differently in different situations, then it would be much harder. You have to make sure both a/ that you never hard link a file that is under a hard-linked directory to somewhere outside of that hard-linked directory and b/ that you never hard link a directory that contains a file which is hard-linked to somewhere outside that directory. The first is probably quite manageable. The second is essentially the cycle-detection problem. /technical-content humour SUN used to advertise: The network is the computer However I think we have all come to realise that the network is the network, and the computer is the computer. Now Hans wants to tells that The directory is the file but I don't find it any more convincing than SUN's message... /humour opinion If you really want to change traditional Unix semantics, I would suggest you get rid of hard-links. They really are more trouble than they are worth, and discarding them makes this whole issue moot. /opinion NeilBrown
Re: reiser4 plugins
On Monday 04 July 2005 05:42, Hans Reiser wrote: Horst von Brand wrote: Right. But, /proc started somewhere, didn't it? Sun. No, plan 9. Actually, I heard a lecture on implementing /proc in BSD at least 5 years earlier than plan 9. It seems to me that what you are all arguing about is whether it is better to be at the front of the herd, or in the middle, or in the back. I don't think this one can be resolved, except that I would suggest that Linux, by virtue of the kernel configuration functionality, has space enough for persons of all three inclinations, if we can just leave each other be a bit. Hans
Re: reiser4 plugins
malcolm wrote: On Monday 04 July 2005 05:42, Hans Reiser wrote: Horst von Brand wrote: Right. But, /proc started somewhere, didn't it? Sun. No, plan 9. Actually, I heard a lecture on implementing /proc in BSD at least 5 years earlier than plan 9. http://library.n0i.net/linux-unix/art-unix-programming/plan9.html says Some Plan 9 ideas have been absorbed into modern Unixes, particularly the more innovative open-source versions. FreeBSD has a /proc file system modeled exactly on that of Plan 9 that can be used to query or control running processes. Can you cite the other viewpoint? It seems to me that what you are all arguing about is whether it is better to be at the front of the herd, or in the middle, or in the back. I don't think this one can be resolved, except that I would suggest that Linux, by virtue of the kernel configuration functionality, has space enough for persons of all three inclinations, if we can just leave each other be a bit. Hans
Re: reiser4 plugins
On Sun, Jul 03, 2005 at 08:42:24PM -0700, Hans Reiser wrote: Right. But, /proc started somewhere, didn't it? Sun. No, plan 9. Almost on the right track, it was v8, two steps before plan9. But that's just the process-part of procfs, not the big mess we have now - that part is pretty much linux-only.
Re: reiser4 plugins
Horst von Brand wrote: David Masover [EMAIL PROTECTED] wrote: David Weinehall wrote: On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: David Weinehall wrote: GNOME and KDE run on operating systems that run other kernels than Linux, hence they have to implement their own userland VFS anyway. Adding this to the Linux kernel won't help them one bit, unless we can magically convince Sun to add it to Solaris, all different BSD:s to add it to their kernels, etc. Not going to happen. An effort to get GNOME and KDE to unify their VFS:s would be far more benificial, Than what? Creating a unified VFS which I can access from Bash, and which obsoletes both GNOME and KDE's VFSes except in their presentation? On one of the platforms that they support, yes. But only for kernels newer than 2.6.yy... So they'd still have to have their own VFS for 2.4.xx, 2.6.xx (xx yy), FreeBSD, OpenBSD, Solaris, etc... Right. But, /proc started somewhere, didn't it? Sun. I have the feeling that other systems will duplicate it if it's good. Linux copied here. So what? Even if they don't, it would be more beneficial to me How, exactly? Go back and read. Besides, /your/ convenience isn't the only thing that matters... Nor yours. Just because you can't get your mind around a concept doesn't mean that it's a bad concept, and that no one else can use it. and probably most Linux users Most Linux users don't use experimental filesystems at all... Actually, they do -- ext3 was experimental once. ReiserFS was very experimental once. Please stop bashing it just because it's new/experimental. to have metafs supported in both GNOME and KDE, even if they still need an emulation layer to support other systems. So Gnome and KDE get larger (and thus slower) for everybody. Smaller (and thus faster) on supported systems, otherwise exactly the same, but maybe a little more modular, which is good. Besides, Gnome and KDE will have to agree on the formats involved first, which is /exactly/ what is supposed to be impossible unless this stuff is implemented in the kernel... I never said that, but for one thing, whether they do or not, it's nice if my shell and my editor and all the other things that I use don't have to agree on anything to manipulate the formats involved. This is not just about GNOME/KDE. It is about GNOME/KDE not developing an additional layer, that you wouldn't like anyway, that cannot be accessed from anything except GNOME/KDE.
Re: reiser4 plugins
Horst von Brand wrote: Kevin Bowen [EMAIL PROTECTED] wrote: [...] So, for instance, if I want to grab all mp3s with Artist Paul Oakenfold and change the genre to techno (can you do that?), I can use Beagle's search tool to find all mp3s by Oakenfold, but to change the genre, I have to use some separate tool to manipulate id3 tags, Yes, this is basically what I was getting at, although I was thinking more in terms of the reiser5/6/whatever set theoretic semantics, which, from my point of view at least, reiser4 is simply the first step towards building the enabling infrastructure of. But the fact that reiser4 semantics + trivial shell scripting enables, as illustrated by David, the rough equivalent of set-theoretic semantics, demonstrates how reiser4 is in fact a step in this direction. I don't see any trivial shell scripting, just need for a plethora of magic extract-this-or-that-metadata programs. Which can very well work exactly the same on independent files, no need to shove junk /into/ the indexed files. moment the case of system-wide or network-wide shared data, I.e., something like 90% of the use of Linux here. OK. users needs. In fact, I believe there is currently a JSR in progress to develop a more sophisticated Java packaging model. Presumably based on ReiserFS 4, which then has to be ported to whatever platform you want to run Java on ASAP? Great for you! Wait a bit, and you'll get what you want then, even across the board! No, obviously that's not what I was saying. But the need for these kinds of domain-specific packaging/metadata formats, each requiring their own tools to work with, would be alleviated if there were simply a more powerful filesystem semantic. *Please explain HOW*!! The domain-specific formats /will/ stay (if nothing else, because the /domains/ have /specific/ needs), the special tools to work with them /will/ have to be written exactly the same (because of the above). All for use on /one/ non-standard filesystem. One filesystem that exists right now. /proc, as people have made clear, was originally implemented on *one* non-standard Unix. Others then copied it. I see no reason why if /meta is a good idea, and programs start using it, that other FSes won't implement it. Plus exactly the same stuff to work on sane filesystems. sane -- I assume you mean traditional. Well, yeah. So? How big a program do you need to create a new interface to an existing system? Consider: BitTorrent has at least three interfaces that I can remember at the moment. btdownloadgui.py, btdownloadcurses.py, btdownloadheadless.py. Are you saying that btdownloadgui.py was so much work if you start from btdownloadcurses.py that there should be no GTK library, because it takes so much work to convert existing curses apps to GTK? Remember, even if I have GTK installed, I can still get to my curses apps, and they may not even know that GTK exists. Even if I have /meta enabled, I can still use my POSIX apps, and they may not even know that /meta exists. So-called 'database-filesystems' ARE coming, whether from Microsoft, Apple, or us. IBM did it, long ago (AS/400 and OS/400 ring a bell?). And is now slowly moving away from it (and other structured filesystems), AFAICS, towards (guess what!) POSIX and Linux... Linux over POSIX, I think. Because Linux is big and open-source, not beacuse POSIX is so great. Again, Linux is what it is today in large part because We have to get $FEATURE, because if not, $COMPETITOR will win on us! arguments have no traction. I agree with that, at least. But Linux is also what it is today in large part because I don't need $FEATURE, so it should be kept out isn't a particularly good argument either.
RE: reiser4 plugins
Hi, As long as we're being pedantic, not plan 9, research edition 8. The original /proc file system was done by Tom Killian (I early misattributed it to Ritchie) for research edition 8, back in '84. (See http://blogs.sun.com/roller/page/eschrock/20040625 for history) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hans Reiser Sent: Sunday, July 03, 2005 8:42 PM To: Horst von Brand Cc: David Masover; David Weinehall; Markus Törnqvist; Douglas McNaught; Hubert Chan; Kyle Moffett; [EMAIL PROTECTED]; Lincoln Dale; Gregory Maxwell; Jeff Garzik; Christoph Hellwig; Andrew Morton; linux-kernel@vger.kernel.org; ReiserFS List Subject: Re: reiser4 plugins Horst von Brand wrote: Right. But, /proc started somewhere, didn't it? Sun. No, plan 9.
Re: reiser4 plugins
David Masover [EMAIL PROTECTED] wrote: Horst von Brand wrote: David Masover [EMAIL PROTECTED] wrote: David Weinehall wrote: On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: David Weinehall wrote: [...] Even if they don't, it would be more beneficial to me How, exactly? Go back and read. I have. And have seen /no/ benefit to you. Except, of course, the benefit accrued from some magic in Linus' kernel, by which all format differences go in a puff of smoke if they are implemented inside it, and furthermore all userland gets rebuilt to use the kernel's way overnight. Sorry, I'm quite sceptical on this count. Sure, for complete outsiders it may look that way: Something gets implemented in the kernel, soon many applications are using it, everything is peachy. Except that before there have been /long/ (even bitter) discussions on if/how to do it, what the kernel should do (if anything) and what the application's responsibility should be. Many times it has ended with Linus slamming on the table and saying no way, or we do it /this/ way. Then, /when/ this is clear, it gets into the kernel and applications. And you only have ever seen the last phase... which is smooth and fast. Here, the how should it be done, if at all discussion has barely started. And AFAICS there are plenty of ways of getting 95% of the advantages /without/ touching the kernel, so that should be the path to follow. Nobody has shown any real evidence at all for this not being so. Besides, /your/ convenience isn't the only thing that matters... Nor yours. Sure enough. Just because you can't get your mind around a concept doesn't mean that it's a bad concept, and that no one else can use it. I /do/ get my mind around the concept, it /is/ sometimes useful. It /can/ be done without the kernel, and so /should/ be done that way. That is all. and probably most Linux users Most Linux users don't use experimental filesystems at all... Actually, they do -- ext3 was experimental once. Right. And ext2, and ext, and xiafs, and ... So? When they were experimental, only a handful of utter fools commited their data to them. ReiserFS was very experimental once. See above. Please stop bashing it just because it's new/experimental. Sorry? to have metafs supported in both GNOME and KDE, even if they still need an emulation layer to support other systems. So Gnome and KDE get larger (and thus slower) for everybody. Smaller (and thus faster) on supported systems, Sorry, but just because some $DISTRO gives ReiserFS4 as an /option/ doesn't mean they will have everything duplicated as For ReiserFS4 and For other filesystems. My assertion stands, until there are ReiserFS4-only distributions. otherwise exactly the same, but maybe a little more modular, which is good. Right, having to cope with different ways of representing metadata could induce better code organization. But to get there isn't painless either... Besides, Gnome and KDE will have to agree on the formats involved first, which is /exactly/ what is supposed to be impossible unless this stuff is implemented in the kernel... I never said that, You (and others) have told us time and again that each of them have their own incompatible ways of handling metadata, and that only if this was handled in the kernel they would make use of a common way of managing it... but for one thing, whether they do or not, it's nice if my shell and my editor and all the other things that I use don't have to agree on anything to manipulate the formats involved. Please, read what you just said. Everybody (kernel somewhat included) will have to agree on the format of the metadata. This is not just about GNOME/KDE. It is about GNOME/KDE not developing an additional layer, that you wouldn't like anyway, How do you know what I would or would not like? that cannot be accessed from anything except GNOME/KDE. So Gnome and KDE agree on some secret formats that only they can handle? Behind their user's backs? As open source? And happily shoot their feet by not giving users much wanted access to said data in decent ways? I see it more on the lines of libmetadata.so (or such), which is used by Gnome, KDE, and whatever else wishes to do so. Even your custom-tailored shell or cat(1). Or just some convention that metadata on ~/my/funky/file resides in ~/.metadata/.my/.funky/.file/metadata. Hey, you could (almost) do that today, by wrappers (perhaps even aliases, or shell functions) around the relevant commands! No kernel at all involved! Not even a miserable library!! -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431
Re: reiser4 plugins
Kevin Bowen [EMAIL PROTECTED] wrote: [...] So, for instance, if I want to grab all mp3s with Artist Paul Oakenfold and change the genre to techno (can you do that?), I can use Beagle's search tool to find all mp3s by Oakenfold, but to change the genre, I have to use some separate tool to manipulate id3 tags, Yes, this is basically what I was getting at, although I was thinking more in terms of the reiser5/6/whatever set theoretic semantics, which, from my point of view at least, reiser4 is simply the first step towards building the enabling infrastructure of. But the fact that reiser4 semantics + trivial shell scripting enables, as illustrated by David, the rough equivalent of set-theoretic semantics, demonstrates how reiser4 is in fact a step in this direction. I don't see any trivial shell scripting, just need for a plethora of magic extract-this-or-that-metadata programs. Which can very well work exactly the same on independent files, no need to shove junk /into/ the indexed files. moment the case of system-wide or network-wide shared data, I.e., something like 90% of the use of Linux here. OK. 90% of *what* exactly? 90% of what machines deal with, or 90% of what humans interact with? Both. Most use here is in computer labs, where most is shared via NFS (readonly), plus user data mounted read-write. users needs. In fact, I believe there is currently a JSR in progress to develop a more sophisticated Java packaging model. Presumably based on ReiserFS 4, which then has to be ported to whatever platform you want to run Java on ASAP? Great for you! Wait a bit, and you'll get what you want then, even across the board! No, obviously that's not what I was saying. But the need for these kinds of domain-specific packaging/metadata formats, each requiring their own tools to work with, would be alleviated if there were simply a more powerful filesystem semantic. *Please explain HOW*!! The domain-specific formats /will/ stay (if nothing else, because the /domains/ have /specific/ needs), the special tools to work with them /will/ have to be written exactly the same (because of the above). All for use on /one/ non-standard filesystem. Plus exactly the same stuff to work on sane filesystems. The kernel is *not* a magic piece of software that solves the problem of world hunger if you just can figure out what strange extension to force onto Linus' kernel version. Clearly forcing reiser on the world is a non-starter, but try extending your imagination a little bit to a future in which there's a 'new POSIX' specifying a set-theoretic filesystem model. Sorry, I just don't see any need of shoving Oracle into the kernel. It leads a quite confortable life in userland. So-called 'database-filesystems' ARE coming, whether from Microsoft, Apple, or us. IBM did it, long ago (AS/400 and OS/400 ring a bell?). And is now slowly moving away from it (and other structured filesystems), AFAICS, towards (guess what!) POSIX and Linux... Again, Linux is what it is today in large part because We have to get $FEATURE, because if not, $COMPETITOR will win on us! arguments have no traction. Who gets there first will determine who gets to make the rules. So what? Let /them/ make the mistakes and pay for them, and learn how to do it /right/, even if later! -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
David Masover [EMAIL PROTECTED] wrote: David Weinehall wrote: On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: David Weinehall wrote: GNOME and KDE run on operating systems that run other kernels than Linux, hence they have to implement their own userland VFS anyway. Adding this to the Linux kernel won't help them one bit, unless we can magically convince Sun to add it to Solaris, all different BSD:s to add it to their kernels, etc. Not going to happen. An effort to get GNOME and KDE to unify their VFS:s would be far more benificial, Than what? Creating a unified VFS which I can access from Bash, and which obsoletes both GNOME and KDE's VFSes except in their presentation? On one of the platforms that they support, yes. But only for kernels newer than 2.6.yy... So they'd still have to have their own VFS for 2.4.xx, 2.6.xx (xx yy), FreeBSD, OpenBSD, Solaris, etc... Right. But, /proc started somewhere, didn't it? Sun. I have the feeling that other systems will duplicate it if it's good. Linux copied here. Even if they don't, it would be more beneficial to me How, exactly? Besides, /your/ convenience isn't the only thing that matters... and probably most Linux users Most Linux users don't use experimental filesystems at all... to have metafs supported in both GNOME and KDE, even if they still need an emulation layer to support other systems. So Gnome and KDE get larger (and thus slower) for everybody. Besides, Gnome and KDE will have to agree on the formats involved first, which is /exactly/ what is supposed to be impossible unless this stuff is implemented in the kernel... -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, ChileFax: +56 32 797513
Re: reiser4 plugins
Le Fri, 01 Jul 2005 04:08:20 -0500, David Masover [EMAIL PROTECTED] a écrit : Hi all, Metafs also avoids having to patch tar. It's assumed that legacy backup systems can always avoid metafs and still catch almost everything important, and certainly everything they already do catch. With a hybrid or an entirely new backup system, we could catch everything, including any new ACL-like animals that people invent. As an administrator, a way to perfectly save and restore the content of file systems is a requirement. Crashes do happen, so do migration to new disk subsystems, system cloning, etc, etc. If restoration can be done with the standard tools you find on any live rescue CD, it's even better. All what's probably needed is some readable-writable non-confusing (fully POSIX compliant) view that's guaranteed to contain all existing raw data and metadata). BR, Pierre.
Re: reiser4 plugins
Pierre Etchemaïté wrote: Le Fri, 01 Jul 2005 04:08:20 -0500, David Masover [EMAIL PROTECTED] a écrit : Hi all, Metafs also avoids having to patch tar. It's assumed that legacy backup systems can always avoid metafs and still catch almost everything important, and certainly everything they already do catch. With a hybrid or an entirely new backup system, we could catch everything, including any new ACL-like animals that people invent. As an administrator, a way to perfectly save and restore the content of file systems is a requirement. Crashes do happen, so do migration to new disk subsystems, system cloning, etc, etc. I would suggest a plugin which produces a dump of the active FS. It's how XFS seems to do it, after all, only you don't need the commandline tool. If restoration can be done with the standard tools you find on any live rescue CD, it's even better. Any good XFS-enabled CD will have xfsdump. Similarly, any good Reiser4-enabled CD will have whatever we feel like putting there. Can ACLs currently be backed up with any other tool than some sort of dump? All what's probably needed is some readable-writable non-confusing (fully POSIX compliant) view that's guaranteed to contain all existing raw data and metadata). The danger here is that things beyond POSIX must either be encoded into the file somehow. If we add a thumbnail property to a certain type of file, and the file format itself doesn't allow for any extra data, we can either choose to have a separate view entirely for backup so that the backup view (which forces the metadata into the file anyway) is separate, or we can choose to accept that existing backup tools will catch existing attributes that they were programmed to expect, and all new stuff can be handled by some sort of dump. I say some sort of dump because a separate backup view, because it makes some files unusable except for transport, is effectively a dump, only shown as a tree, instead of a single file. But as long as we're doing that, we may as well provide an actual dump plugin so that one can do gzip /metas/some/where/.../dump /mnt/some/backup/medium. It'd be more efficient than tar for that sort of thing. Both are useful, though -- the tree is better for rsync backups, if you're either using /meta on the other end or are planning to only restore to a /meta enabled system. I think we should do both ways.
Re: reiser4 plugins
Horst von Brand wrote: Kevin Bowen [EMAIL PROTECTED] wrote: [...] The desire amongst users for ubiquitous metadata is very real - the current wave of desktop search products and technologies demonstrates this - Just like each previous claim this /must/ be the next cool technology!, also called later the dotcom crash... Have you used this technology? For that matter, some of these claims have been true. The Internet, despite the dotcom crash, is now a requirement -- few people would use an OS with no networking support on a desktop PC. Same goes for GUIs. Both of these technologies had to have at least some amount of kernel support, both were once weird and experimental, and both are now in demand by users and likely to be for some time. If we're right, and searching is the next best thing, then it would be a good idea to get started now. If we're wrong, what's the harm? (Assuming /meta, of course. I accept that the issues with file-as-directory may never be solved to the point where we can do it at the kernel's FS/VFS level.) Google Desktop Search et al allow for querying on metadata, but actually *acting* on the results of those queries requires that they be exposed via first class primitives which can be manipulated with arbitrary tools, not via some proprietary userland api which only one tool ever actually implements. Could you please explain in plain english? The only part I get out is propietary API, and I don't see anybody advocating such here. I don't understand it much, but I think the point being made is that while desktop searches may be able to search metadata (not sure how they do this -- their own plugins or just full-text searching?), they cannot write to said metadata. To change metadata on a file, one currently needs to use a proprietary userland api which is only implemented by one tool. So, for instance, if I want to grab all mp3s with Artist Paul Oakenfold and change the genre to techno (can you do that?), I can use Beagle's search tool to find all mp3s by Oakenfold, but to change the genre, I have to use some separate tool to manipulate id3 tags, which may or may not exist in an easily scriptable form. I know there's a decent tool for id3 tags, but there's so many kinds of metadata and so many tools out there that the user can't possibly memorize them all. This is why I use the zipfile example, despite its flaws. Now, the arbitrary tools bit is about how one could do this kind of thing with file-as-dir or /meta. With file-as-dir, it might be as simple as: $ for i in `magical-find-tool artist='Paul Oakenfold'`; do echo 'techno' $i/.../genre; done Or even $ echo 'techno' /.../search/artist='Paul Oakenfold'/*/.../genre It doesn't get too much more complicated for /meta, which has a nice side effect of not killing things like tar and generally not messing with POSIX. One could also use vim, emacs, or whatever else to edit this metadata, instead of being limited to one program. This is especially nice considering you might have to look up which one program to use. As to the case of system-wide shared files, there is already a mechanism to prevent users from inappropriately annotating files that don't belong to them: file permissions. And who says that a normal user isn't allowed to annotate each and every file with its purpose or something else? I can very well imagine a system in which users (say students in a Linux class) want to do so... on a shared machine. Or users having a shared MP3 or photograph or ... collection, with individual notes on each. Or even developers wanting to annotate source code files with their comments, but leave them read-only (or have them under SCM). File-as-dir is actually pretty nice for this, I think. Since the annotations appear as files, if the app supports it, the users can each store individual annotations, world-readable, but only user-writable. With something like Gnome-VFS, I doubt there's even a mechanism whereby such annotations can be shared to begin with. But this is just a thought experiment atm, don't pick it apart too brutally...
backup (was Re: reiser4 plugins)
Dmitry Torokhov wrote: On 6/30/05, Theodore Ts'o [EMAIL PROTECTED] wrote: On Mon, Jun 27, 2005 at 04:12:38PM -0500, David Masover wrote: Streamload cannot warrant and does not guarantee, and You should not expect, that all of Your private communications and other personal information will never be disclosed in ways not otherwise described in this Privacy Policy. gpg. Was in my upload script to begin with. I keep my key written many times on a single hidden CD. So long as the isofs can be read, at least one of the copies should be usable. They don't have any billing information on me. If they charge me for something, I'll cancel my account. If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice. Does not look like it: 3c. No encryption and/or steganography for the purpose of circumventing Streamload's rules. ... For example, if you'd like to encrypt something for an extra sense of security and privacy, please feel free to do so. However, when these tools are used solely for the purpose of circumventing Streamload's rules, as determined by the sole discretion of Streamload, the files will be deleted. Trasnlation: They really, honestly, have no way at all of knowing if my GPG-encrypted 450 meg tarball violates their terms. They have legally given themselves the right to delete it just because they think it might violate their terms, but it wouldn't make sense for them to delete a file called backup.tar.bz2.gpg when there are no doubt a lot of other encrypted files on their servers. It wouldn't make sense because they would either have to delete such files completely at random, or they'd have to delete all of them at once. Further translation: This was probably put here to keep the lawyers happy. They don't want to be held liable for illegal stuff on their servers, for example, so they have at least spelled out, in legalese, that illegal stuff (and probably a few other categories) is not allowed on their servers, and that encyrpting it doesn't magically make it legal. This kind of stuff makes lawyers happy, even if it *practically* does nothing at all. Regardless, I feel reasonably safe with a backup there. If I ever start to feel unsafe, I can always back up the most critical stuff to gmail, and create a few permanent copies on DVDs. But I feel considerably safer with Streamload than with Gmail, because using Gmail for my own custom backup app directly violates their terms of service. I think we should officially kill this particular thread.
Re: reiser4 plugins
Zan Lynx wrote: On Thu, 2005-06-30 at 14:47 -0400, Horst von Brand wrote: [snip] [snip-some-more] Structured files are fine when they're small but quickly become disgusting as they get bigger. Either you slowly rewrite the whole thing or you fast save by writing new sections to the end of it that replace existing sections. That's how you end up with documents that contain deleted information that was supposed to be confidential. Doesn't OpenOffice already do something similar with zipfiles? I remember noticing huge differences in save times after adding large images to a presentation versus just changing some text. So, it's obviously not repacking each image every time I save. But, as the internal text is all XML, I doubt it's leaving deleted sections lying around. Having the filesystem track each part for you and then creating a structured file when needed (and without needing to remember to run a special tool) is a far better idea. (reiser4 doesn't do this but its a possible idea) Been discussed to death, and chances are, Reiser4 will do this with plugins. I imagine it being implemented in a more general way, though. For example, a meta-file which, when read, produces a zipfile/tarball/dump of the directory it belongs to. Then, to send such a document via email, you just navigate to the directory where it lives while looking for files to attach, only from /meta/vfs as a base, then attach the meta-file. The zipfile gets built automatically. To simplify the process, all you *really* need is a button on all FS navigation windows (Nautilus, Open/Save dialogs, etc) to switch back and forth between the meta view and the normal view. That, or a separate pane. This way, instead of patching the mail client to support all possible transformations you'd want to do before sending, or patching all applications to use zip (and any other transformation a user might want to do), or forcing the user to run another app, you automatically get *all* available transformations, in a way that could potentially look as smooth as adding zip support directly to the app. Another advantage to file-as-directory is being able to access all the file's meta-data through a single channel: file names and text data. It removes the need for chmod, chown, touch, getxattr, chacl and all the rest of the special tools needed to work with Unix files. It also makes it far easier to add new meta-data in the future, because no new tools have to be written to use it. All good points except the last one. New tools vs new plugins? People already know how to write the tools... So that's two reasons to bother. That, and the possible paradigm shift. Navigating to the /meta dir for a file could be the commandline equivalent of right-clicking on the file. Of course, to some people, paradigm shifts are a reason *not* to bother...
Re: reiser4 plugins
Horst von Brand wrote: Kevin Bowen [EMAIL PROTECTED] wrote: and think about it just limited to a user's home directory, and the storage and organization of actual *data* (as opposed to system files). Who is to say what is data and what is system files? And if you are limiting yourself to /user/ data, why not have the /user/ decide how they want to organize it, and give them the tools? That's exactly what the idea of exporting the metadata as files within a file-as-dir is intended to do: let the user decide how to manage metadata. They have the tools -- anything which can interact with a file. This includes users who never start X. The desire amongst users for ubiquitous metadata is very real - the current wave of desktop search products and technologies demonstrates this - Just like each previous claim this /must/ be the next cool technology!, also called later the dotcom crash... How does that analogy hold? An example of a technology which is actually in demand is similar to the tech bubble exactly how? but search is really only the lowest-hanging fruit of this new way of looking at data. Application-layer solutions like Beagle, That works without screwing up the whole system, AFAIU. If you define works as implements a proprietary layer which can't be used naturally by the same tools UNIX users have used for years to manage data. And who says that a normal user isn't allowed to annotate each and every file with its purpose or something else? I can very well imagine a system in which users (say students in a Linux class) want to do so... on a shared machine. Or users having a shared MP3 or photograph or ... collection, with individual notes on each. Or even developers wanting to annotate source code files with their comments, but leave them read-only (or have them under SCM). This same argument could be used to attack the idea of group permissions -- that groups of users might have conflicting goals. Implementing metadata in userspace via bundled files has the same drawback. If you're sysadmining a multiuser reiser4 box, and your users are able to modify the metadata of files they don't own, then you go to sysadmin purgatory. Bingo! And thus goes much of the supposed advantage of this nonsense. You seem to be implying that on a Reiser4 filesystem used by multiple users that people other than file owners will be about to apply arbitrary metadata to any object. It seems to me that adding metadata to a file-as-dir will require the same permissions as writing to the file itself, giving Reiser4 *zero* disadvantage against other multiuser systems. What gives you the impression that adding metadata to a file-as-dir is any harder to secure against modification from other users than the files themselves are in the first place? [I see that /opponents/ are accused here of lack of imagination, while I see that the /proponents/ lack imagination... or perhaps just real-world experience.] Huh? The idea of arbitrary metadata isn't just with annotating MP3 files. One seemingly important consideration would be the flexible implementation of add-on mechanisms, like security data, that could be implemented with zero changes to the filesystem. And you get to use existing tools to deal with new and arbitrary interfaces instead of having to deal with GUI garbage just to interact with metadata. other way; OpenOffice /has/ structured files, XML inside zipped files, Java also uses zip files for its structuring needs, etc), or are ideas that And as a Java developer, I can tell you that the wide consensus is that this solution is half-assed and insufficient for both developers and users needs. In fact, I believe there is currently a JSR in progress to develop a more sophisticated Java packaging model. Presumably based on ReiserFS 4, which then has to be ported to whatever platform you want to run Java on ASAP? Great for you! Wait a bit, and you'll get what you want then, even across the board! Are unable to differentiate between a statement that a given solution is non-ideal and a suggestion that everything be done in a way that requires Reiser4? Someone mentions that the current solution isn't quite meeting all of their needs, and you respond with rhetoric accusing them of wanting everything to depend on Reiser4 and bloat the kernel. How is this productive?
Re: reiser4 plugins
Hubert Chan wrote: On Thu, 30 Jun 2005 08:29:56 +0200, David Weinehall [EMAIL PROTECTED] said: On Thu, Jun 30, 2005 at 12:33:10AM -0400, Hubert Chan wrote: It's sort of like the way web servers handle index.html, for those who think it's a stupid idea. (Of course, some people may still think it's a stupid idea... ;-) ) And guess what? That's handled on the web server level (userland), not by the file system. So different web servers can handle it differently (think index.html.sv, index.html.zh, index.php, etc). From the web *browser*'s point of view, it is handled by the filesystem (which is provided by the various servers). The browser doesn't care how or where the data is stored. It just requests a file, and gets some data back. So the browser doesn't have to check for http://www.example.com/, get a failure (trying to read a directory), check for http://www.example.com/index.html, etc. In this way, the web server controls (which I think takes the place of the filesystem in this case) what gets shown (index.html.sv, etc.), instead of the leaving it up to the browser. Somewhat flawed analogy, though. After the protocol definition, the browser proper will take any URL that the protocol handler likes, which is why file:// works. After the domain, the http/https handler will take any URL at all, except for maybe some character set issues. So assuming the server is compatible with itself, we don't have to worry about whether the browser supports going to a directory and having it behave as a file (index.html behavior), or going to a file and having it behave as a directory (as some scripts do -- I've seen urls that look like http://example.com/foo.cgi/bar.html) Among protocols that behave more like filesystems, such as FTP, you can't really pull the same tricks. When an FTP client asks for a directory, it's probably asking for a directory listing, and would be quite surprised to find a file there -- or the user would when binary data floods their terminal. I *think* this is how FTP works, but I haven't used it in years, except through a web browser. I still get the feeling that even a web browser expects an FTP file to behave as a file and an FTP directory to behave as a directory. But I'm also pretty sure that FTP would be much more receptive to file-as-directory than your average sysadmin would. For one, breaking tar is unforgivable, and the only ways I can think of fixing that issue are shaky at best when you consider how many apps might do things oh-so-slightly different than tar.
Re: reiser4 plugins
Hubert Chan wrote: On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro [EMAIL PROTECTED] said: I'm confused. Can someone on one of these lists enlighten me? How is directories as files logically any different than putting all data into .data files and making all files directories (yes you would need some sort of special handling for files that were really called .data). Then it's just a matter of deciding what happens when you call open and stat on one of these files? Logically, I don't think there is a difference. A filesystem that doesn't support file-as-dir could implement the same functionality that way. [1] In fact, that's essentially what MacOS X/NeXTSTEP does with its bundle format -- it's just a regular directory with regular files inside. I, personally, would hate it if everything in my /bin suddenly became a directory, mainly because everything would stop working. Is that the kind of thing you're suggesting? I'm a little confused about the .data idea, I guess. But we could have a whole new set of system calls that treat things as magic, and if files as directories is as cool as many people think, apps will start using the new api. If not, they won't and the new api can be deprecated. File-as-dir doesn't require new system calls (that I know of), which is the whole point of the idea. Existing programs can edit the strange new attributes without being modified. That is indeed the point, but scroll down. The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... [...] People like Horst (and probably others, who are less vocal), I think, don't think that it's even worth trying it out because they don't see any major advantages. Or at least they think that the potential negatives outweigh the potential positives. I respect that they have different opinions, but I of course disagree and attempt to convince them otherwise. Did the /meta (metafs) idea get killed while I was out? Using that approach, your potential negatives are that apps which crawl the entire FS tree, starting at /, with hardcoded apps for /proc and /sys, are now broken -- but then, /sys already broke them once, so I don't particularly care if we break them again. Potential positives? I think even just because we like the idea is enough, because it doesn't break anything and doesn't really affect anyone who doesn't use it. Maybe there are coding standards, but I think others are working that out now.
Re: reiser4 plugins
Wow. Good that you noticed it. It sounds like they have the business model of an intelligence agency. Hans Theodore Ts'o wrote: If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice. - Ted
Re: reiser4 plugins
Dmitry Torokhov wrote: If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice. Does not look like it: 3c. No encryption and/or steganography for the purpose of circumventing Streamload's rules. ... For example, if you'd like to encrypt something for an extra sense of security and privacy, please feel free to do so. However, when these tools are used solely for the purpose of circumventing Streamload's rules, as determined by the sole discretion of Streamload, the files will be deleted. Oops, sorry, got that wrong, ok, it is more that they provide this service and the terms are at their convenience not anyone else's.
Re: reiser4 plugins
David Weinehall wrote: On Wed, Jun 29, 2005 at 04:58:20PM +0300, Markus Törnqvist wrote: On Wed, Jun 29, 2005 at 09:50:27AM -0400, Douglas McNaught wrote: I'll just note that the applications bundled as directories stuff on MacOS/NextStep is done completely in userspace--as far as the kernel is concerned, Mail.app is a regular directory. The file manager handles recognition and invocation of application bundles (and there is an 'open' shell command that does the same thing). Note that MacOS has the monopoly on what they ship, Linux has a motherload of file managers and window systems and all. What pisses me off is the fact that Gnome and friends implement their own incompatible-with-others VFS's and automounters and stuff. Surely supporting this in the kernel and extending the LSB to require this is the best step to take without infringing anyone's freedom as such. *still pissed off about having to hassle an automatic mount* GNOME and KDE run on operating systems that run other kernels than Linux, hence they have to implement their own userland VFS anyway. Adding this to the Linux kernel won't help them one bit, unless we can magically convince Sun to add it to Solaris, all different BSD:s to add it to their kernels, etc. Not going to happen. An effort to get GNOME and KDE to unify their VFS:s would be far more benificial, Than what? Creating a unified VFS which I can access from Bash, and which obsoletes both GNOME and KDE's VFSes except in their presentation? FreeDesktop is doing a lot of work to make GNOME, KDE, and other DE:s interoperate as much as possible. Support their initiative instead of trying to get a monstrosity (albeit a very cool one, conceptually) into the kernel. Sure, it could be made to work, but not without dropping our Unixness. (I'm talking about the metafs (/meta) idea, which isn't nearly as much a monstrocity, and doesn't kill our unixness, it enhances it.)
Re: reiser4 plugins
Horst von Brand wrote: Markus Törnqvist [EMAIL PROTECTED] wrote: [...] Note that MacOS has the monopoly on what they ship, Linux has a motherload of file managers and window systems and all. Yep. Part of what is nice about it, too ;-) What pisses me off is the fact that Gnome and friends implement their own incompatible-with-others VFS's and automounters and stuff. Then get them to agree on a common framework! They are trying hard to get other parts of the GUI work well together, so this isn't far off in wishfull thinking land. Unfortunately, this leaves bash out in the cold, as usual. Kernel-based automounter works with Bash. Why can't GNOME/KDE use the kernel's one as at least one backend, even if they support others?
Re: reiser4 plugins
It was always the expectation that users would want to have one mountpoint with the files having metafiles as files, and another with the same files but strictly posix, and then their apps can use whichever they have the power to understand. Hans David Masover wrote: Hubert Chan wrote: On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro [EMAIL PROTECTED] said: I'm confused. Can someone on one of these lists enlighten me? How is directories as files logically any different than putting all data into .data files and making all files directories (yes you would need some sort of special handling for files that were really called .data). Then it's just a matter of deciding what happens when you call open and stat on one of these files? Logically, I don't think there is a difference. A filesystem that doesn't support file-as-dir could implement the same functionality that way. [1] In fact, that's essentially what MacOS X/NeXTSTEP does with its bundle format -- it's just a regular directory with regular files inside. I, personally, would hate it if everything in my /bin suddenly became a directory, mainly because everything would stop working. Is that the kind of thing you're suggesting? I'm a little confused about the .data idea, I guess. But we could have a whole new set of system calls that treat things as magic, and if files as directories is as cool as many people think, apps will start using the new api. If not, they won't and the new api can be deprecated. File-as-dir doesn't require new system calls (that I know of), which is the whole point of the idea. Existing programs can edit the strange new attributes without being modified. That is indeed the point, but scroll down. The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... [...] People like Horst (and probably others, who are less vocal), I think, don't think that it's even worth trying it out because they don't see any major advantages. Or at least they think that the potential negatives outweigh the potential positives. I respect that they have different opinions, but I of course disagree and attempt to convince them otherwise. Did the /meta (metafs) idea get killed while I was out? Using that approach, your potential negatives are that apps which crawl the entire FS tree, starting at /, with hardcoded apps for /proc and /sys, are now broken -- but then, /sys already broke them once, so I don't particularly care if we break them again. Potential positives? I think even just because we like the idea is enough, because it doesn't break anything and doesn't really affect anyone who doesn't use it. Maybe there are coding standards, but I think others are working that out now.
Re: reiser4 plugins
Hans Reiser wrote: It was always the expectation that users would want to have one mountpoint with the files having metafiles as files, and another with the same files but strictly posix, and then their apps can use whichever they have the power to understand. It was never in the early betas I tried :( I'm proposing (or re-proposing) that the /meta mountpoint be strictly for accessing meta-files, with no intention of eventually using this by default. Furthermore, /meta should follow POSIX anyway, mostly -- no file-as-dir there, either, although you still wouldn't want to use tar on it. With only a few patches, using /meta could be almost as convenient as using file/..metas/foo, and it completely kills the file-as-directory flamewar -- everybody's happy. Or so I thought. It seems that the people arguing for file-as-directory are ignoring /meta, and the people arguing against it are arguing against all meta-files, saying that the good things about meta-files don't justify the risks of file-as-dir. Only once did I see someone bring up /meta. I don't like the idea of having /meta have file-as-dir and everything as we originally wanted, because then we've duplicated the interface. To read the *contents* of /foo, I can either cat /foo or cat /meta/foo. I'd hate to have the POSIX mountpoint still lying around if no one's using it, even more than I hate the idea of foo being in two places for no good reason. Is there a technical (performance?) reason that my approach is wrong? (metas go in /meta, files go in /, and everything feels POSIX-y) Hans David Masover wrote: Hubert Chan wrote: On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro [EMAIL PROTECTED] said: I'm confused. Can someone on one of these lists enlighten me? How is directories as files logically any different than putting all data into .data files and making all files directories (yes you would need some sort of special handling for files that were really called .data). Then it's just a matter of deciding what happens when you call open and stat on one of these files? Logically, I don't think there is a difference. A filesystem that doesn't support file-as-dir could implement the same functionality that way. [1] In fact, that's essentially what MacOS X/NeXTSTEP does with its bundle format -- it's just a regular directory with regular files inside. I, personally, would hate it if everything in my /bin suddenly became a directory, mainly because everything would stop working. Is that the kind of thing you're suggesting? I'm a little confused about the .data idea, I guess. But we could have a whole new set of system calls that treat things as magic, and if files as directories is as cool as many people think, apps will start using the new api. If not, they won't and the new api can be deprecated. File-as-dir doesn't require new system calls (that I know of), which is the whole point of the idea. Existing programs can edit the strange new attributes without being modified. That is indeed the point, but scroll down. The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) What's the locking issue? I think that was more about transactions... [...] People like Horst (and probably others, who are less vocal), I think, don't think that it's even worth trying it out because they don't see any major advantages. Or at least they think that the potential negatives outweigh the potential positives. I respect that they have different opinions, but I of course disagree and attempt to convince them otherwise. Did the /meta (metafs) idea get killed while I was out? Using that approach, your potential negatives are that apps which crawl the entire FS tree, starting at /, with hardcoded apps for /proc and /sys, are now broken -- but then, /sys already broke them once, so I don't particularly care if we break them again. Potential positives? I think even just because we like the idea is enough, because it doesn't break anything and doesn't really affect anyone who doesn't use it. Maybe there are coding standards, but I think others are working that out now.
Re: reiser4 plugins
David Masover wrote: Hans Reiser wrote: It was always the expectation that users would want to have one mountpoint with the files having metafiles as files, and another with the same files but strictly posix, and then their apps can use whichever they have the power to understand. It was never in the early betas I tried :( Yes, well, there are a lot of things missing in functionality still from what was, and still is, in the plan. Inheritance, files not listed by readdir, etc., lots of things need coding. We have done the hard stuff first though, so these other things will mostly not be a lot of code Hans
Re: reiser4 plugins
Hubert Chan wrote: On Tue, 28 Jun 2005 15:22:55 -0500, David Masover [EMAIL PROTECTED] said: Come to think of it, that changes the proposal a bit. You need a different way to access the meta-dir for files than for directories, if we're going to support /meta/vfs. No biggie: /meta/vfs/file/home/bob/sue.mpg/acl /meta/vfs/dir/home/bob/acl What if /home has an extended attribute named bob? Then is /meta/vfs/dir/home/bob a file or a directory? Ah, you caught it. I knew it didn't feel right. So /meta/vfs/file and /meta/vfs/dir are worthless. Ok, there has to be a delimiter of some kind. That, or there has to be a way we can pick up front how deep we're going. Or, someone else has to come up with a better idea. Or, start out with the assumption that we're talking about metadata, instead of the other way around. I don't like delimiters because the way we usually deal with these is escaping them when we need to. For instance, if I want to actually pass a literal character to some command, I can wrap it in ' characters or add a \ to the front of it. But we're expecting this to be simpler than that. A program trying to access the metas for some file shouldn't have to worry about escaping. Last time we tried to come up with a delimiter, we ended up with ... and ..metas as possible candidates, and ..metas was in the source. As unlikely as it is to hit either of those, I still don't like them. Programs would mostly use the inodes anyway, and users would mostly use the /meta/vfs interface, but it's still not pretty. But, the only alternative I can think of now is /meta/vfs/2/home/bob is a file and /meta/vfs/3/home/bob is a directory. That's not something we want our users to have to do, especially considering we want to be able to have metadata of metadata. That, or we can try something like: /meta/vfs/#/home/#/bob is a directory /home/vfs/#/home/bobis a file where we add a delimiter to the /meta dir which identifies where each directory begins, not each set of metadata. To clarify a bit: $ ls /meta/vfs/ mime permissions acls # [snip] $ ls /meta/vfs/#/ proc sys tmp etc home [snip] $ ls /meta/vfs/#/home mime permissions acls # [snip] $ ls /meta/vfs/#/home/#/ bob billy hank sue [snip] This is pretty annoying for the user, though, because it's more verbose. But, at least this way, we can guarentee that no one will kill our delimiter, because it exists in a new namespace we're creating anyway. That is, if we have to, the old xattrs stuff can go in its own folder, and so even if some app currently depends on a '#' app, it won't kill our delimiter. Now, if only there was an easier delimiter to type... Ok, since file-as-dir was going to do this anyway, I think the way to go is probably the original '...' delimiter. It's safer, because it stays in /meta, but it's dangerous, because someone might actually try to create a '...' file on an existing system. But, I think we can live with introducing a delimiter, more than we can live with uglifying the /meta/vfs interface, especially because touch ... wouldn't make the system blow up, you just couldn't easily get metadata for ...
Re: reiser4 plugins
Hubert Chan wrote: On Tue, 28 Jun 2005 15:59:03 -0400, Kyle Moffett [EMAIL PROTECTED] said: On Jun 28, 2005, at 13:51:04, Hubert Chan wrote: I don't know if VFS is the right place for it, but I agree that it would be good to make it accessible to all filesystems. That's somewhat of a contradiction in terms. The whole point of the VFS is to hold all of the things that multiple filesystems want to share :-D. VFS provides a common interface to the filesystem. I don't think metafs needs any VFS changes. It may be able to get by without making changes to the VFS, and if so, it shouldn't touch the VFS. It should just be its own separate filesystem. I imagine most of it could be implemented by a FUSE filesystem. could, yes. should, no. I'll refer you to my HURD comment. That's not to say that none of this should be userspace, just that some of it most certainly *never* needs to touch userspace, such as cryptocompress. I'm not guessing that you wanted to make it FUSE, I just want to be pre-emptive here. FUSE will NOT work well for this. Maybe we just need better regular applications? You mean patch them all so that they understand and can edit xattr/substreams/etc.? The file-as-dir interface is meant to avoid having to do that. metafs also avoids having to patch all the applications by exposing them as regular files. Metafs also avoids having to patch tar. It's assumed that legacy backup systems can always avoid metafs and still catch almost everything important, and certainly everything they already do catch. With a hybrid or an entirely new backup system, we could catch everything, including any new ACL-like animals that people invent.
Re: backup (was Re: reiser4 plugins)
Rule #3 from http://www.streamload.com/About/Legal_eng.asp?page=id73# is pretty clear about what applies if you have a trial account (which seems to be what you have since you say you'll cancel your account if they charge you anything): 3. Do not circumvent Freeloader download restrictions. Any attempt to circumvent Freeloader trial account restrictions will result in a permanent banishment from the Streamload community. This includes (3a) files with an invalid or disguised file format; (3b) encryption; (3c) steganography and/or (3d) creating multiple freeloader accounts. You can interpret that whatever way you like, but if you're that cavalier with your data, hey, I'm not sure _I'd_ trust your judgement about which filesystem to trust. :-) - Ted
Re: reiser4 plugins
On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: David Weinehall wrote: GNOME and KDE run on operating systems that run other kernels than Linux, hence they have to implement their own userland VFS anyway. Adding this to the Linux kernel won't help them one bit, unless we can magically convince Sun to add it to Solaris, all different BSD:s to add it to their kernels, etc. Not going to happen. An effort to get GNOME and KDE to unify their VFS:s would be far more benificial, Than what? Creating a unified VFS which I can access from Bash, and which obsoletes both GNOME and KDE's VFSes except in their presentation? On one of the platforms that they support, yes. But only for kernels newer than 2.6.yy... So they'd still have to have their own VFS for 2.4.xx, 2.6.xx (xx yy), FreeBSD, OpenBSD, Solaris, etc... FreeDesktop is doing a lot of work to make GNOME, KDE, and other DE:s interoperate as much as possible. Support their initiative instead of trying to get a monstrosity (albeit a very cool one, conceptually) into the kernel. Sure, it could be made to work, but not without dropping our Unixness. (I'm talking about the metafs (/meta) idea, which isn't nearly as much a monstrocity, and doesn't kill our unixness, it enhances it.) Which would neither need VFS changes nor be dependent on Reiser4 in any way, so I don't see why this thread lives on. Just get down to business and implement this metafs =) Regards: David Weinehall -- /) David Weinehall [EMAIL PROTECTED] /) Northern lights wander (\ // Maintainer of the v2.0 kernel // Dance across the winter sky // \) http://www.acc.umu.se/~tao/(/ Full colour fire (/
Re: backup (was Re: reiser4 plugins)
Theodore Ts'o wrote: Rule #3 from http://www.streamload.com/About/Legal_eng.asp?page=id73# is pretty clear about what applies if you have a trial account (which seems to be what you have since you say you'll cancel your account if they charge you anything): 3. Do not circumvent Freeloader download restrictions. Any attempt to circumvent Freeloader trial account restrictions will result in a permanent banishment from the Streamload community. This includes (3a) files with an invalid or disguised file format; (3b) encryption; (3c) steganography and/or (3d) creating multiple freeloader accounts. You can interpret that whatever way you like, but if you're that cavalier with your data, hey, I'm not sure _I'd_ trust your judgement about which filesystem to trust. :-) Streamload is one possibility. There are others -- gmail and such. This is just what can be had *for*free*. For that matter, the uploading process doesn't require a user to agree to those terms at all. There's a module in CPAN that allows an upload, and never asks me to agree to any terms, nor does it ask me for an account password. I don't have multiple accounts, I don't *use* the encryption to attempt to circumvent the download restrictions, especially considering that said download restrictions don't even come into play until I need a restore. By the time I do need a restore, I'm willing to pay money, as 100 megs a month is certainly not going to be enough.
Re: reiser4 plugins
David Weinehall wrote: On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: David Weinehall wrote: GNOME and KDE run on operating systems that run other kernels than Linux, hence they have to implement their own userland VFS anyway. Adding this to the Linux kernel won't help them one bit, unless we can magically convince Sun to add it to Solaris, all different BSD:s to add it to their kernels, etc. Not going to happen. An effort to get GNOME and KDE to unify their VFS:s would be far more benificial, Than what? Creating a unified VFS which I can access from Bash, and which obsoletes both GNOME and KDE's VFSes except in their presentation? On one of the platforms that they support, yes. But only for kernels newer than 2.6.yy... So they'd still have to have their own VFS for 2.4.xx, 2.6.xx (xx yy), FreeBSD, OpenBSD, Solaris, etc... Right. But, /proc started somewhere, didn't it? I have the feeling that other systems will duplicate it if it's good. Even if they don't, it would be more beneficial to me and probably most Linux users to have metafs supported in both GNOME and KDE, even if they still need an emulation layer to support other systems. I agree that the emulation layer should be common. FreeDesktop is doing a lot of work to make GNOME, KDE, and other DE:s interoperate as much as possible. Support their initiative instead of trying to get a monstrosity (albeit a very cool one, conceptually) into the kernel. Sure, it could be made to work, but not without dropping our Unixness. (I'm talking about the metafs (/meta) idea, which isn't nearly as much a monstrocity, and doesn't kill our unixness, it enhances it.) Which would neither need VFS changes nor be dependent on Reiser4 in any way, so I don't see why this thread lives on. Just get down to business and implement this metafs =) I am trying, at least getting the conceptual glitches ironed out. But, it does need GNOME/KDE VFS changes in order to be sane -- otherwise, GNOME/KDE will duplicate it, creating even more of a mess than we already have. I'm not sure why the thread still lives, but now that I've typed this, I may as well send it...
