Re: Versioning File System for FreeBSD?
On Thu, Jun 25, 2009 at 12:33:23AM +0200, Polytropon wrote: On Wed, 24 Jun 2009 23:57:34 +0200, cpghost cpgh...@cordula.ws wrote: Yep, you're right. I thought about a way to extend the API in a backwards compatible way, but that's not as easy or straight forward as it seems. In fact, it opens a whole can of worms. If the versioned file system isn't also POSIX compatible (where everything happens in HEAD unless specified otherwise), it's practically useless. The question is: Do you want to take versioning support into the file system intendedly? FreeBSD keeps most things on a per-file basis (ordinary files, devices, processes etc.). Versioning can always be added as a separate solution (using versioning systems as separate programs) that does not make any assumptions on the file system used. As you concluded, the file system's complexity would of course grow with those requirements. In addition to your arguments, just imagine how a fsck for such a file system would have to be implemented... Yep. The more I think about it, the less obvious it becomes. IMHO, file versioning a la VMS would be possible (somehow), but everything beyond that (esp. directory versioning) requires a LOT of careful thinking. -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
On Thu, Jun 25, 2009 at 01:59:45AM +0200, Roland Smith wrote: On Wed, Jun 24, 2009 at 11:57:34PM +0200, cpghost wrote: Quite true! I see even more ambiguity here: What about a versioned file pointed to by hard links from two versioned directories? The more I think about it, the more problems I can see. Look e.g. at symbolic links. Or looking from the vc side, what about branches (checking out an older version of a file and editing it). Does it automatically become the new head, or are concurrent branches allowed? And even if the semantics were absolutely sound (can they be?), all this meta data really needs to happen on a block level, e.g. how described in that paper. I really wonder if combining a filesystem and a version control system is a good idea? After a good night's sleep, and rethinking the whole concept, I agree that this is not such a good idea after all. At least not until I fully understand how (directory) versioning actually is supposed to work semantically AND under the hood. I'll stick to subversion (and will try git and hg as well), until I find a better solution. And there's another problem here: what if two processes concurrently save (commit?) the same file, and there's a merging conflict? I'd say that two processes should _never_ open the same file for writing at the same time. Since the contents of the file are opaque to the file system but not to the programs, it is impossible for the filesystem to fix merge conflicts. Right! If you have multiple programs working together only one should write to the file in question. The others should communicate with the writing program via IPC. Serializing file access? Yes, that makes sense. Of course, there would be additional API calls to traverse the list of revisions, to access meta data (properties?, tags?, commit logs?, ...) etc., so that the file system remains manageable. VMS had a filesystem that uses versioning: [http://en.wikipedia.org/wiki/Files-11] I was thinking about this before starting this thread. But file versioning (as opposed to full versioning that also includes directory versioning) is probably relatively easy to implement. At least, its semantics are unambiguous. Indeed. It seems the VMS filesystem just tacks a semicolon and a nummer on to the filename. Yep, that's one way to do it. If you're willing to go to the block level, I could imagine the inode of a versioned file linking to versioned direct / indirect blocks, i.e. one inode linking to more than just one (physical) file. To keep things simple, the inode could link to a circular buffer of N (direct/indirect block links). Those versioned files could also COW-share blocks, but that's nothing conceptual, just an optimization. That would be pure file versioning: directories are linking to the inode, and each inode would potentially refer to N revisions of a file. But if it makes sense or not is something else altogether. Thanks for the great brainstorming. Things are clearer to me now. ;-) Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
On Wed, Jun 24, 2009 at 05:04:22PM +0200, cpghost wrote: Hi, is there anybody working on a versioning file system for FreeBSD right now? I don't care if it is native or a layer, geom-ified, fuse-based, or even if it uses subversion as its backend, as long as it provides some kind of transparent versioning. You could try devel/git? It's not a file system, but a very efficient directory content tracker/version control system. One of the things I use it for is to keep a certain directory in sync between a desktop and a laptop. It works equally well with text or binary files. Or maybe http://www.nongnu.org/libsqlfs/ will do what you want? Haven't tried that, though. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgp2oszWbtvQL.pgp Description: PGP signature
Re: Versioning File System for FreeBSD?
On Wed, Jun 24, 2009 at 06:37:55PM +0200, Roland Smith wrote: On Wed, Jun 24, 2009 at 05:04:22PM +0200, cpghost wrote: Hi, is there anybody working on a versioning file system for FreeBSD right now? I don't care if it is native or a layer, geom-ified, fuse-based, or even if it uses subversion as its backend, as long as it provides some kind of transparent versioning. You could try devel/git? It's not a file system, but a very efficient directory content tracker/version control system. One of the things I use it for is to keep a certain directory in sync between a desktop and a laptop. It works equally well with text or binary files. Yes, that's one possibility. But just like Subversion (which I'm using extensively here), it's not really transparent. Or maybe http://www.nongnu.org/libsqlfs/ will do what you want? Haven't tried that, though. But does it really support versioning? I was actually thinking of a real versioning file system, with an extended POSIX API (yet to be defined), to access all revisions of a file system, just like with Subversion revisions. As an example: opendir(2) would grow an additional and optional argument revision to select either HEAD or some revision of the directory: DIR *dirp; dirp = opendir(/path/to/dir, 0); /* open /path/to/file at HEAD */ dirp = opendir(/path/to/dir); /* same as above, POSIX compat */ dirp = opendir(/path/to/dir, 323); /* open dir at revision 323 */ /* From here on, readdir() would retrieve /path/to/dir entries at the specified revision. */ open(2) could open a file at an earlier revision: FILE *filep; /* open file in HEAD */ filep = open(/path/to/file, O_RDONLY); /* open same path, but at revision 323 */ filep = open(/path/to/file, O_RDONLY, /* 0666 */, 323); unlink(2) would remove an entry from a directory, and bump the revision of the directory. Accessing that path from the new revision wouldn't be possible, but the file would still be there in an earlier revision. Modifying a file would also create new revisions (e.g. at each write(2), or at each close(2), that should be selectable). Of course, there would be additional API calls to traverse the list of revisions, to access meta data (properties?, tags?, commit logs?, ...) etc., so that the file system remains manageable. I didn't try them (yet), but on Linux, there are some experimental versioning file systems like: http://www.ext3cow.com/Welcome.html http://tux3.org/ and there's a (unmaintained?) FUSE file system at: http://wayback.sourceforge.net/ all of which differ in the way POSIX API should be extended and the semantics for versioning. But there's apparently nothing yet in the works for FreeBSD. Perhaps some layer on top of existing file systems, or an extension of UFS/FFS that stores versioning meta data directly at the block level? Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
On Wed, Jun 24, 2009 at 07:59:18PM +0200, cpghost wrote: open(2) could open a file at an earlier revision: FILE *filep; s/FILE */int /; -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
On Wed, Jun 24, 2009 at 07:59:18PM +0200, cpghost wrote: On Wed, Jun 24, 2009 at 06:37:55PM +0200, Roland Smith wrote: On Wed, Jun 24, 2009 at 05:04:22PM +0200, cpghost wrote: Hi, is there anybody working on a versioning file system for FreeBSD right now? I don't care if it is native or a layer, geom-ified, fuse-based, or even if it uses subversion as its backend, as long as it provides some kind of transparent versioning. You could try devel/git? It's not a file system, but a very efficient directory content tracker/version control system. One of the things I use it for is to keep a certain directory in sync between a desktop and a laptop. It works equally well with text or binary files. Yes, that's one possibility. But just like Subversion (which I'm using extensively here), it's not really transparent. What is? If you have to extend the API like you propose below, all programs that want to use that feature have to be changed. So if you're going around changing your program, why not have it interface to an existing revision control that you are already familiar with? That seems a lot easier that tacking revision control onto a filesystem! Git is very good at efficiently storing the differences between commits. And every copy of a directory under git control is a full-blown repository, so you can experiment with a copy without fear of fouling up your precious repository. I was actually thinking of a real versioning file system, with an extended POSIX API (yet to be defined), to access all revisions of a file system, just like with Subversion revisions. As an example: opendir(2) would grow an additional and optional argument revision to select either HEAD or some revision of the directory: DIR *dirp; dirp = opendir(/path/to/dir, 0); /* open /path/to/file at HEAD */ dirp = opendir(/path/to/dir); /* same as above, POSIX compat */ dirp = opendir(/path/to/dir, 323); /* open dir at revision 323 */ /* From here on, readdir() would retrieve /path/to/dir entries at the specified revision. */ open(2) could open a file at an earlier revision: FILE *filep; /* open file in HEAD */ filep = open(/path/to/file, O_RDONLY); /* open same path, but at revision 323 */ filep = open(/path/to/file, O_RDONLY, /* 0666 */, 323); There is some ambiguity here. Does 323 refer to a single file, or to the state of its parent directory? If changing a file doesn't update the version of its parent directory, then why have version numbers for directories? On the other hand, if changing a file updates the revision for the file and its parent directory, the revision for the root directory will increase quite rapidly! unlink(2) would remove an entry from a directory, and bump the revision of the directory. Accessing that path from the new revision wouldn't be possible, but the file would still be there in an earlier revision. Modifying a file would also create new revisions (e.g. at each write(2), or at each close(2), that should be selectable). I don't know what you want to do use this for, but a simple trick (used e.g. by Pro/Engineer) is to have your application append a version number after the filename (e.g. foo.prt.1) that is incremented every time the file is saved. This does waste some disk space (or provides redundancy, take your pick). Of course, there would be additional API calls to traverse the list of revisions, to access meta data (properties?, tags?, commit logs?, ...) etc., so that the file system remains manageable. VMS had a filesystem that uses versioning: [http://en.wikipedia.org/wiki/Files-11] I didn't try them (yet), but on Linux, there are some experimental versioning file systems like: http://www.ext3cow.com/Welcome.html http://tux3.org/ and there's a (unmaintained?) FUSE file system at: http://wayback.sourceforge.net/ all of which differ in the way POSIX API should be extended and the semantics for versioning. But there's apparently nothing yet in the works for FreeBSD. Perhaps some layer on top of existing file systems, or an extension of UFS/FFS that stores versioning meta data directly at the block level? Roland -cpghost. -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgpsKGcvW25kv.pgp Description: PGP signature
Re: Versioning File System for FreeBSD?
On Wed, 24 Jun 2009 21:11:25 +0200, Roland Smith rsm...@xs4all.nl wrote: VMS had a filesystem that uses versioning: [http://en.wikipedia.org/wiki/Files-11] That's the first thing that came into my mind when reading this message. See LOGIN.COM;1 and then rm -rf /*.*;* :-) But it's not had, it's has, because VMS and its file system does still exist. -- Polytropon From Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
* cpghost cpgh...@cordula.ws [2009-06-24 17:04 +0200]: Hi, is there anybody working on a versioning file system for FreeBSD right now? - I don't know how fare along hammerfs is in being ported to FreeBSD. But from what I have heard, feature-wise, it might be something that meets your needs. An alternative could be ZFS with its snapshot ability. -- I have no signature! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
On Wed, Jun 24, 2009 at 09:11:25PM +0200, Roland Smith wrote: Yes, that's one possibility. But just like Subversion (which I'm using extensively here), it's not really transparent. What is? If you have to extend the API like you propose below, all programs that want to use that feature have to be changed. So if you're going around changing your program, why not have it interface to an existing revision control that you are already familiar with? That seems a lot easier that tacking revision control onto a filesystem! Yep, you're right. I thought about a way to extend the API in a backwards compatible way, but that's not as easy or straight forward as it seems. In fact, it opens a whole can of worms. If the versioned file system isn't also POSIX compatible (where everything happens in HEAD unless specified otherwise), it's practically useless. Git is very good at efficiently storing the differences between commits. And every copy of a directory under git control is a full-blown repository, so you can experiment with a copy without fear of fouling up your precious repository. That's true as well. I'm not very familiar with git (as opposed to subversion), but I clearly see its advantages. I was actually thinking of a real versioning file system, with an extended POSIX API (yet to be defined), to access all revisions of a file system, just like with Subversion revisions. As an example: opendir(2) would grow an additional and optional argument revision to select either HEAD or some revision of the directory: DIR *dirp; dirp = opendir(/path/to/dir, 0); /* open /path/to/file at HEAD */ dirp = opendir(/path/to/dir); /* same as above, POSIX compat */ dirp = opendir(/path/to/dir, 323); /* open dir at revision 323 */ /* From here on, readdir() would retrieve /path/to/dir entries at the specified revision. */ open(2) could open a file at an earlier revision: FILE *filep; /* open file in HEAD */ filep = open(/path/to/file, O_RDONLY); /* open same path, but at revision 323 */ filep = open(/path/to/file, O_RDONLY, /* 0666 */, 323); There is some ambiguity here. Does 323 refer to a single file, or to the state of its parent directory? If changing a file doesn't update the version of its parent directory, then why have version numbers for directories? On the other hand, if changing a file updates the revision for the file and its parent directory, the revision for the root directory will increase quite rapidly! Quite true! I see even more ambiguity here: What about a versioned file pointed to by hard links from two versioned directories? And even if the semantics were absolutely sound (can they be?), all this meta data really needs to happen on a block level, e.g. how described in that paper. unlink(2) would remove an entry from a directory, and bump the revision of the directory. Accessing that path from the new revision wouldn't be possible, but the file would still be there in an earlier revision. Modifying a file would also create new revisions (e.g. at each write(2), or at each close(2), that should be selectable). I don't know what you want to do use this for, but a simple trick (used e.g. by Pro/Engineer) is to have your application append a version number after the filename (e.g. foo.prt.1) that is incremented every time the file is saved. This does waste some disk space (or provides redundancy, take your pick). Yes, that's always possible. But that would defeat transparency. And there's another problem here: what if two processes concurrently save (commit?) the same file, and there's a merging conflict? Of course, there would be additional API calls to traverse the list of revisions, to access meta data (properties?, tags?, commit logs?, ...) etc., so that the file system remains manageable. VMS had a filesystem that uses versioning: [http://en.wikipedia.org/wiki/Files-11] I was thinking about this before starting this thread. But file versioning (as opposed to full versioning that also includes directory versioning) is probably relatively easy to implement. At least, its semantics are unambiguous. I didn't try them (yet), but on Linux, there are some experimental versioning file systems like: http://www.ext3cow.com/Welcome.html http://tux3.org/ and there's a (unmaintained?) FUSE file system at: http://wayback.sourceforge.net/ all of which differ in the way POSIX API should be extended and the semantics for versioning. But there's apparently nothing yet in the works for FreeBSD. Perhaps some layer on top of existing file systems, or an extension of UFS/FFS that stores versioning meta data directly at the block level? -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail
Re: Versioning File System for FreeBSD?
On Wed, Jun 24, 2009 at 11:26:50PM +0200, Morten Grunnet Buhl wrote: * cpghost cpgh...@cordula.ws [2009-06-24 17:04 +0200]: Hi, is there anybody working on a versioning file system for FreeBSD right now? - I don't know how fare along hammerfs is in being ported to FreeBSD. But from what I have heard, feature-wise, it might be something that meets your needs. Ah, yes, that's interesting too. I'll investigate it. Thanks for the hint! An alternative could be ZFS with its snapshot ability. IMHO, ZFS snapshots are a little bit too coarse versioning for what I have in mind. I'm seeking something more precise. Not entirely a subversion-on-filesystem replacement, but rather close, if possible. Thanks, -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
On Wed, 24 Jun 2009 23:57:34 +0200, cpghost cpgh...@cordula.ws wrote: Yep, you're right. I thought about a way to extend the API in a backwards compatible way, but that's not as easy or straight forward as it seems. In fact, it opens a whole can of worms. If the versioned file system isn't also POSIX compatible (where everything happens in HEAD unless specified otherwise), it's practically useless. The question is: Do you want to take versioning support into the file system intendedly? FreeBSD keeps most things on a per-file basis (ordinary files, devices, processes etc.). Versioning can always be added as a separate solution (using versioning systems as separate programs) that does not make any assumptions on the file system used. As you concluded, the file system's complexity would of course grow with those requirements. In addition to your arguments, just imagine how a fsck for such a file system would have to be implemented... -- Polytropon From Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Versioning File System for FreeBSD?
On Wed, Jun 24, 2009 at 11:57:34PM +0200, cpghost wrote: Quite true! I see even more ambiguity here: What about a versioned file pointed to by hard links from two versioned directories? The more I think about it, the more problems I can see. Look e.g. at symbolic links. Or looking from the vc side, what about branches (checking out an older version of a file and editing it). Does it automatically become the new head, or are concurrent branches allowed? And even if the semantics were absolutely sound (can they be?), all this meta data really needs to happen on a block level, e.g. how described in that paper. I really wonder if combining a filesystem and a version control system is a good idea? unlink(2) would remove an entry from a directory, and bump the revision of the directory. Accessing that path from the new revision wouldn't be possible, but the file would still be there in an earlier revision. Modifying a file would also create new revisions (e.g. at each write(2), or at each close(2), that should be selectable). I don't know what you want to do use this for, but a simple trick (used e.g. by Pro/Engineer) is to have your application append a version number after the filename (e.g. foo.prt.1) that is incremented every time the file is saved. This does waste some disk space (or provides redundancy, take your pick). Yes, that's always possible. But that would defeat transparency. Why? Larger number==later file. How more transparent could it be? And there's another problem here: what if two processes concurrently save (commit?) the same file, and there's a merging conflict? I'd say that two processes should _never_ open the same file for writing at the same time. Since the contents of the file are opaque to the file system but not to the programs, it is impossible for the filesystem to fix merge conflicts. If you have multiple programs working together only one should write to the file in question. The others should communicate with the writing program via IPC. Of course, there would be additional API calls to traverse the list of revisions, to access meta data (properties?, tags?, commit logs?, ...) etc., so that the file system remains manageable. VMS had a filesystem that uses versioning: [http://en.wikipedia.org/wiki/Files-11] I was thinking about this before starting this thread. But file versioning (as opposed to full versioning that also includes directory versioning) is probably relatively easy to implement. At least, its semantics are unambiguous. Indeed. It seems the VMS filesystem just tacks a semicolon and a nummer on to the filename. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgpEkArHkSmA6.pgp Description: PGP signature
Re: versioning file system
years ago I used a VAX VMS system that automatically saved file versions whenever a file name was clobbered. I've seen wrapper scripts for vi to accomplish the same but it would be nice if someone had actually implemented something at the file system level to do this. Anyone know of anything like that for FreeBSD? Nope, not that I know of... I'm not sure you'll find a lot of people who like this unless they were accustomed to VMS back however long ago. Ken ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: versioning file system
- Original Message - From: Kenneth Culver [EMAIL PROTECTED] To: David Bear [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Monday, June 23, 2003 5:23 PM Subject: Re: versioning file system years ago I used a VAX VMS system that automatically saved file versions whenever a file name was clobbered. I've seen wrapper scripts for vi to accomplish the same but it would be nice if someone had actually implemented something at the file system level to do this. Anyone know of anything like that for FreeBSD? Nope, not that I know of... I'm not sure you'll find a lot of people who like this unless they were accustomed to VMS back however long ago. Ken It's something Hans Reiser has on his wishlist for ReiserFS, but it's way off, and I think ReiserFS is the only project contemplating it. Would recomend you look at RCS in the meantime. Adam ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
