Re: file as a directory
Back in November 2004, I suggested on the linux-kernel and reiserfs lists that the Reiser4 architecture could allow us to abolish the unnatural naming distinction between directories/files/parts-of-file (i.e. to unify naming within-file-system and within-file naming) in an efficient way. I suggested that one way of doing that would be to extend XPath-like selection syntax above the (XML) file level. (See the archive of the discussion starting at http://www.ussg.iu.edu/hypermail/linux/kernel/0411.3/0044.html Wed Nov 24 2004 - 04:21:13 EST.) ITworld now has an interesting article by Sean McGrath on a very similar idea, mentioning the XML OASIS Open Document Format. What do you think? Peter Foldiak Here it is: -- ITworld http://www.itworld.com/AppDev/1246/nls_ebizbooks050510/ Books/chapters and directories/files - dichotomies considered harmful ITworld.com, Ebusiness in the Enterprise 5/9/05 Sean McGrath, ITworld.com The distinction between a full book and a mere chapter of a book, is a source of endless fascination for incurable information modellers like me. Obviously, at the logical level, the distinction is driven by the content itself. A book is a complete unit of stuff. A chapter, is a sub-division within the complete book. At the physical level, however, technology starts to influence the book/chapter distinction. A chapter boundary, for Microsoft Word users or Open Office users, is likely to be influenced by how big the underlying file gets. Large files take longer to load and get increasingly slower to work with in typical word processing environments. Our decisions about where to draw the chapter boundaries are influenced to some extent by technology limitations. If the physical constraints are not allowed to dictate the boundaries for chapters, then we can end up resorting to file naming conventions to split the content into manageable chunks e.g. chapter1_a, chapter1_b and so on. We might then decide to keep things clean by introducing a subdirectory for each chapter, putting the sub-chapters tidily away in their own little compartments. All is well with the world. Or is it? This is where things get interesting from an information management perspective. A full unit of work - a book - has now been split into bits that are navigable through a directory structure and bits that are navigable through an application. The result? You can use off-the-shelf tools to navigate your way through the directories. You can see the overall structure of the book by simply looking at the directory structure as a hierarchy. You can see that chapter 1 has a number of sub-chapters. However, that is as far as you can go. To dig any further into the structure of chapter 1, section A, you need to launch the editing application. What a pity. Why is it, that we have this hard and fast dichotomy between directory structure and file structure? Why is it that file system exploring utilities need to stop in their tracks when they hit things called 'files'? As you have probably noticed, this artificial split can be breached in certain circumstances, at least to some extent. Graphics file formats are a good example. Many file system exploring tools know about, say, JPEG files and can display thumbnails of their contents. That is a start in the right direction but I think it needs to go a lot further if the artificial directory/file distinction is to be eradicated. Let us go back to the book example. Let us use Microsoft's OLE technology as an analogy. With OLE you can embed one thing in another. So for example, you can embed an Excel spreadsheet into a Word document file. Now, in your head, take that further. Imagine a world in which the file system explorer is the top level application. It manages a single, humungous file on the disk into which you embed documents, spreadsheets, databases etc. Each think you embed into the explorer can itself embed other things to any depth required. In such a world, directories/files have merged into one abstraction. The book author does not have to introduce artificial segmentation of the book into separate entities. In such a world, filenames become something of an oddity. What do you need filenames for? You would only really need a filename at the point where you decided to exchange information between systems A and B. Moreover, once the package of data is pasted into System B's file system explorer at some suitable point, the filename would be thrown away. Sounds interesting wouldn't you say? So why don't we have systems that work like that? There are, as ever, many reasons. One reason which was an issue some years ago, is ceasing to be an issue very quickly now. Obviously, in order to show the structure of a file a file system explorer needs to look inside the file format. If the file format is proprietary, then we can do nothing. Enter XML-based file formats like the OASIS Open Document Format[1]. The day is coming when file system explorers will be able to do for
Re: file as a directory
I agree with the below in that sometimes you want to see a collection of stuff as one file, and sometimes you want to see it as a tree, and that file format browsers can be integrated into file system browsers to look seamless to users. A quibble: A name is just a means to select a file; he is completely wrong to think that file browsers will eliminate filenames. Hans Peter Foldiak wrote: Back in November 2004, I suggested on the linux-kernel and reiserfs lists that the Reiser4 architecture could allow us to abolish the unnatural naming distinction between directories/files/parts-of-file (i.e. to unify naming within-file-system and within-file naming) in an efficient way. I suggested that one way of doing that would be to extend XPath-like selection syntax above the (XML) file level. (See the archive of the discussion starting at http://www.ussg.iu.edu/hypermail/linux/kernel/0411.3/0044.html Wed Nov 24 2004 - 04:21:13 EST.) ITworld now has an interesting article by Sean McGrath on a very similar idea, mentioning the XML OASIS Open Document Format. What do you think? Peter Foldiak Here it is: -- ITworld http://www.itworld.com/AppDev/1246/nls_ebizbooks050510/ Books/chapters and directories/files - dichotomies considered harmful ITworld.com, Ebusiness in the Enterprise 5/9/05 Sean McGrath, ITworld.com The distinction between a full book and a mere chapter of a book, is a source of endless fascination for incurable information modellers like me. Obviously, at the logical level, the distinction is driven by the content itself. A book is a complete unit of stuff. A chapter, is a sub-division within the complete book. At the physical level, however, technology starts to influence the book/chapter distinction. A chapter boundary, for Microsoft Word users or Open Office users, is likely to be influenced by how big the underlying file gets. Large files take longer to load and get increasingly slower to work with in typical word processing environments. Our decisions about where to draw the chapter boundaries are influenced to some extent by technology limitations. If the physical constraints are not allowed to dictate the boundaries for chapters, then we can end up resorting to file naming conventions to split the content into manageable chunks e.g. chapter1_a, chapter1_b and so on. We might then decide to keep things clean by introducing a subdirectory for each chapter, putting the sub-chapters tidily away in their own little compartments. All is well with the world. Or is it? This is where things get interesting from an information management perspective. A full unit of work - a book - has now been split into bits that are navigable through a directory structure and bits that are navigable through an application. The result? You can use off-the-shelf tools to navigate your way through the directories. You can see the overall structure of the book by simply looking at the directory structure as a hierarchy. You can see that chapter 1 has a number of sub-chapters. However, that is as far as you can go. To dig any further into the structure of chapter 1, section A, you need to launch the editing application. What a pity. Why is it, that we have this hard and fast dichotomy between directory structure and file structure? Why is it that file system exploring utilities need to stop in their tracks when they hit things called 'files'? As you have probably noticed, this artificial split can be breached in certain circumstances, at least to some extent. Graphics file formats are a good example. Many file system exploring tools know about, say, JPEG files and can display thumbnails of their contents. That is a start in the right direction but I think it needs to go a lot further if the artificial directory/file distinction is to be eradicated. Let us go back to the book example. Let us use Microsoft's OLE technology as an analogy. With OLE you can embed one thing in another. So for example, you can embed an Excel spreadsheet into a Word document file. Now, in your head, take that further. Imagine a world in which the file system explorer is the top level application. It manages a single, humungous file on the disk into which you embed documents, spreadsheets, databases etc. Each think you embed into the explorer can itself embed other things to any depth required. In such a world, directories/files have merged into one abstraction. The book author does not have to introduce artificial segmentation of the book into separate entities. In such a world, filenames become something of an oddity. What do you need filenames for? You would only really need a filename at the point where you decided to exchange information between systems A and B. Moreover, once the package of data is pasted into System B's file system explorer at some suitable point, the filename would be thrown away. Sounds interesting wouldn't you say? So why don't we have systems that work like that? There are, as ever, many reasons. One
Re: file as a directory
On Tue, 10 May 2005 10:39:23 BST, Peter Foldiak said: Back in November 2004, I suggested on the linux-kernel and reiserfs lists that the Reiser4 architecture could allow us to abolish the unnatural naming distinction between directories/files/parts-of-file (i.e. to unify naming within-file-system and within-file naming) in an efficient way. I suggested that one way of doing that would be to extend XPath-like selection syntax above the (XML) file level. I believe the consensus was that this needs to happen at the VFS layer, not the FS level. The next step would be designing an API for this - what would the VFS present to userspace, and in what way, and how would backward combatability be maintained? pgpKIGxXIFQdb.pgp Description: PGP signature
Re: file as a directory
On Tue, 2005-05-10 at 16:14, [EMAIL PROTECTED] wrote: On Tue, 10 May 2005 10:39:23 BST, Peter Foldiak said: Back in November 2004, I suggested on the linux-kernel and reiserfs lists that the Reiser4 architecture could allow us to abolish the unnatural naming distinction between directories/files/parts-of-file (i.e. to unify naming within-file-system and within-file naming) in an efficient way. I suggested that one way of doing that would be to extend XPath-like selection syntax above the (XML) file level. I believe the consensus was that this needs to happen at the VFS layer, not the FS level. The next step would be designing an API for this - what would the VFS present to userspace, and in what way, and how would backward combatability be maintained? But can it be done efficiently above the file system level?? As far as I understand, Reiser4 has this nice tree structure, which means that the part of file selection could be done with almost no extra effort, you just attach additional names to inside nodes of the tree, so the same tree can be used to store the whole object, and part of the same tree can be used to select the object part. Right? If you do this above the file system level, I don't think it would have such an efficient implementation. Or would it? Peter
.reiserfs_priv directory removal
This is probably an odd question. I am running a 2.6.5 kernel using reiserfs version 3.6.13. I have two reiserfs filesystems. I need to mount these filesystems under a 2.4.22 kernel with reiserfs version 3.6.2. Then another system mounts them in 2.6.5 and then they are brought back to the original 2.4.22. The reiserfs filesystems were originally created under 2.4.22 using the 3.6.2 tools. When mounted under the original 2.4.22 system a directory appears in the root of the filesystem '.reiserfs_priv'. From doing some research I understand that this directory is for storing xarg and acl information for the partition. My problem is it appears that reiserfs 3.6.2 under kernel 2.4.22 does not require this hidden directory structure. What I would like to be able to do is delete this entire structure while still booted in my 2.6.5 based system so that when the filesystems are later mounted in 2.4.22 the directory structure no longer exists. I won't need to use those filesystems anymore under 2.6.5 after deleting those directories. Pretty much I want to delete them and then unmount the partitions. I can't seem to find a way to do this using user space tools. I tried different mount options as well with no success. Any suggestions would be greatly appreciated. Thanks, Nick
REMOVE UNSUBSCRIBE
UNSUBSCRIBE REMOVE
Re: .reiserfs_priv directory removal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nicholas DeClario wrote: This is probably an odd question. I am running a 2.6.5 kernel using reiserfs version 3.6.13. I have two reiserfs filesystems. I need to mount these filesystems under a 2.4.22 kernel with reiserfs version 3.6.2. Then another system mounts them in 2.6.5 and then they are brought back to the original 2.4.22. The reiserfs filesystems were originally created under 2.4.22 using the 3.6.2 tools. When mounted under the original 2.4.22 system a directory appears in the root of the filesystem '.reiserfs_priv'. From doing some research I understand that this directory is for storing xarg and acl information for the partition. My problem is it appears that reiserfs 3.6.2 under kernel 2.4.22 does not require this hidden directory structure. What I would like to be able to do is delete this entire structure while still booted in my 2.6.5 based system so that when the filesystems are later mounted in 2.4.22 the directory structure no longer exists. I won't need to use those filesystems anymore under 2.6.5 after deleting those directories. Pretty much I want to delete them and then unmount the partitions. I can't seem to find a way to do this using user space tools. I tried different mount options as well with no success. Any suggestions would be greatly appreciated. The only way to remove this directory under a 2.6 kernel is to rebuild the kernel without extended attributes enabled. The .reiserfs_priv directory is special and hidden. The problem you're running into is similar to mounting an ext3 filesystem as ext2 using a kernel from before ext3 was written. It doesn't know how to use it, and so it just doesn't hide the directory. The 2.6 kernels, when it's possible that the directory will be used (ie: xattrs enabled, read-write), creates the directory if it doesn't exist. So, as soon as you mount the filesystem on another 2.6 system, the directory will be recreated unless you've disabled xattrs at compile time. What's the problem with simply removing the directory once you've mounted the filesystem under your 2.4 kernel? - -Jeff - -- Jeff Mahoney SuSE Labs -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (GNU/Linux) iD8DBQFCgN9ILPWxlyuTD7IRAuT+AJ4vi8Q8Z+bx80A4AXoBpo5wa7zzjQCbBYb2 jVZbGVsMgcZmcwvZ/WVWcVY= =DfF3 -END PGP SIGNATURE-
Re: file as a directory
Peter Foldiak wrote: On Tue, 2005-05-10 at 15:53, Hans Reiser wrote: I agree with the below in that sometimes you want to see a collection of stuff as one file, and sometimes you want to see it as a tree, and that file format browsers can be integrated into file system browsers to look seamless to users. A quibble: A name is just a means to select a file; he is completely wrong to think that file browsers will eliminate filenames. Yes, even if you think of the whole file system as a single file, you need a way to select the bit you need, and you will use names for that (and whether you call that a filename, a file-part name or an object name doesn't really matter). The thing that interests me most is the difference (if any) between giving a stream of bytes an opaque name e.g. Chapter 1 of my book.sxw versus giving a stream of bytes a query expression that can also be considered an opaque name e.g. /book/chapter[1] This is what the Russell/Frege descriptive theory of proper names applied to storage systems in a sense[1]. I've written about this stuff before on ITWorld (warning: chatty prose style ahead): Fractals, Self Similarity, and the Whimsical Boundaries of XML Documents http://www.itworld.com/nl/xml_prac/04252002/ A study in XML culture and evolution http://www.itworld.com/nl/ebiz_ent/03252003/ [1] http://en.wikipedia.org/wiki/Proper_name Sean
Re: file as a directory
Peter Foldiak wrote: On Tue, 2005-05-10 at 16:14, [EMAIL PROTECTED] wrote: On Tue, 10 May 2005 10:39:23 BST, Peter Foldiak said: Back in November 2004, I suggested on the linux-kernel and reiserfs lists that the Reiser4 architecture could allow us to abolish the unnatural naming distinction between directories/files/parts-of-file (i.e. to unify naming within-file-system and within-file naming) in an efficient way. I suggested that one way of doing that would be to extend XPath-like selection syntax above the (XML) file level. I believe the consensus was that this needs to happen at the VFS layer, not the FS level. The next step would be designing an API for this - what would the VFS present to userspace, and in what way, and how would backward combatability be maintained? But can it be done efficiently above the file system level?? As far as I understand, Reiser4 has this nice tree structure, which means that the part of file selection could be done with almost no extra effort, you just attach additional names to inside nodes of the tree, so the same tree can be used to store the whole object, and part of the same tree can be used to select the object part. Right? If you do this above the file system level, I don't think it would have such an efficient implementation. Or would it? Peter The tree structure Peter speaks of is a storage layer entity, and so I think Peter's argument is not correct, but what Reiser4 also has is a plugin architecture, and it would be much easier to code it if we use the plugin architecture. Hans
Re: file as a directory
Sean McGrath wrote: The thing that interests me most is the difference (if any) between giving a stream of bytes an opaque name e.g. Chapter 1 of my book.sxw versus giving a stream of bytes a query expression that can also be considered an opaque name e.g. /book/chapter[1] What is an opaque name?
Re: kernel BUG() hit during PCI testing
HI, On Tue, May 10, 2005 at 12:25:01PM -0400, Jeff Mahoney was heard to remark: Linas Vepstas wrote: Hi, I just hit a kernel BUG() during pci testing of 2.6.11.8. The goal of the testing was to temporarily disable a PCI slot containing a SCSI controller. I think I permanently killed the PCI slot; i/o died, and shortly after I hit the BUG(). See below. The goal is, of course, to have the kernel keep on trooping even if the SCSI controller dies out from under it; returning -EIO to user apps accessing the failed file system is acceptable. --linas io-falcons:~ # dmesg -bash: /bin/dmesg: Input/output error (Above is the normal message when a file system returns -EIO to user space; I expect to see these kinds of messages if the block device under the file system fails. Then, a second later I got the crash: io-falcons:~ # io-falcons:~ # io-falcons:~ # cpu 0x0: Vector: 700 (Program Check) at [c001ffe73740] pc: c0138b48: .write_ordered_chunk+0xa4/0x100 lr: c01392f4: .write_ordered_buffers+0x348/0x364 sp: c001ffe739c0 msr: 90029032 current = 0xc003fe6d5030 paca= 0xc0547000 pid = 942, comm = reiserfs/0 kernel BUG in submit_ordered_buffer at fs/reiserfs/journal.c:616! Hi Linas - This one is on my radar among several others of the same type. What's happening is that somehow buffers are getting dirtied despite not being uptodate. They're getting allowed to be put back into the write cycle which is totally invalid, so they're getting caught rather than being written to disk. ext3 has similar problems, but tends to handle them as buffer errors rather than BUGs. I'm investigating whether or not these errors could occur outside individual filesystems. OK, if I don't get busy with other things, I'll look more closely at this as well. Maybe :-/Meanwhile here's the dmesg output between the pci outage and the crash. Don't know if this will be useful. If you want any kind of tracing turned on, let me know. --linas 4sym0: suspicious SCSI data while resetting the BUS. 4sym0: dp1,d15-8,dp0,d7-0,rst,req,ack,bsy,sel,atn,msg,c/d,i/o = 0x7ff, expecting 0x100 5sym0: SCSI BUS has been reset. 4sym0:8:0: HOST RESET operation timed-out. (Yes, that's because the PCI bus is down at this point ... this message is normal.) 6scsi: Device offlined - not ready after error recovery: host 0 channel 0 id 8 lun 0 6scsi: Device offlined - not ready after error recovery: host 0 channel 0 id 8 lun 0 6scsi: Device offlined - not ready after error recovery: host 0 channel 0 id 8 lun 0 3scsi0 (8:0): rejecting I/O to offline device 3scsi0 (8:0): rejecting I/O to offline device 3scsi0 (8:0): rejecting I/O to offline device 3scsi0 (8:0): rejecting I/O to offline device 3Buffer I/O error on device sda4, logical block 743 4lost page write due to I/O error on sda4 3scsi0 (8:0): rejecting I/O to offline device 3Buffer I/O error on device sda4, logical block 744 4lost page write due to I/O error on sda4 3scsi0 (8:0): rejecting I/O to offline device 3Buffer I/O error on device sda4, logical block 745 4lost page write due to I/O error on sda4 3Buffer I/O error on device sda4, logical block 746 4lost page write due to I/O error on sda4 3Buffer I/O error on device sda4, logical block 747 4lost page write due to I/O error on sda4 3Buffer I/O error on device sda4, logical block 748 4lost page write due to I/O error on sda4 3Buffer I/O error on device sda4, logical block 749 4lost page write due to I/O error on sda4 3Buffer I/O error on device sda4, logical block 750 4lost page write due to I/O error on sda4 3Buffer I/O error on device sda4, logical block 751 4lost page write due to I/O error on sda4 3Buffer I/O error on device sda4, logical block 752 4lost page write due to I/O error on sda4 4ReiserFS: sda4: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [20588 360606 0x0 SD] 3scsi0 (8:0): rejecting I/O to offline device 2REISERFS: abort (device sda4): Journal write error in flush_commit_list 2REISERFS: Aborting journal for filesystem on sda4 2kernel BUG in submit_ordered_buffer at fs/reiserfs/journal.c:616! 3scsi0 (8:0): rejecting I/O to offline device 4ReiserFS: sda4: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [20588 88143 0x0 SD] 3scsi0 (8:0): rejecting I/O to offline device 4ReiserFS: sda4: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [20588 243569 0x0 SD] 3scsi0 (8:0): rejecting I/O to offline device 4ReiserFS: sda4: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [20588 287908 0x0 SD] 3scsi0 (8:0): rejecting I/O to offline device 4ReiserFS: sda4: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [20588 294525 0x0 SD] 3scsi0 (8:0):
Re: file as a directory
Hans Reiser wrote: Sean McGrath wrote: The thing that interests me most is the difference (if any) between giving a stream of bytes an opaque name e.g. Chapter 1 of my book.sxw versus giving a stream of bytes a query expression that can also be considered an opaque name e.g. /book/chapter[1] What is an opaque name? By opaque name I mean a name that is purely a label. A name that cannot be interpreted as a query expression. Sean
Re: .reiserfs_priv directory removal
Removing it after rebooting in to 2.4 is definitely a posibility. However, the elegant solution in the scenario here is to do it before hand. I was wondering if it was possible. I guess the most logical route to go would be to have a run-once script that deletes these directories when 2.4 boots. Thanks for you help Jeff. Regards, Nick On 5/10/05, Jeff Mahoney [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The only way to remove this directory under a 2.6 kernel is to rebuild the kernel without extended attributes enabled. The .reiserfs_priv directory is special and hidden. The problem you're running into is similar to mounting an ext3 filesystem as ext2 using a kernel from before ext3 was written. It doesn't know how to use it, and so it just doesn't hide the directory. The 2.6 kernels, when it's possible that the directory will be used (ie: xattrs enabled, read-write), creates the directory if it doesn't exist. So, as soon as you mount the filesystem on another 2.6 system, the directory will be recreated unless you've disabled xattrs at compile time. What's the problem with simply removing the directory once you've mounted the filesystem under your 2.4 kernel? - -Jeff - -- Jeff Mahoney SuSE Labs -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (GNU/Linux) iD8DBQFCgN9ILPWxlyuTD7IRAuT+AJ4vi8Q8Z+bx80A4AXoBpo5wa7zzjQCbBYb2 jVZbGVsMgcZmcwvZ/WVWcVY= =DfF3 -END PGP SIGNATURE-
Re: file as a directory
Sean McGrath wrote: Hans Reiser wrote: Sean McGrath wrote: The thing that interests me most is the difference (if any) between giving a stream of bytes an opaque name e.g. Chapter 1 of my book.sxw versus giving a stream of bytes a query expression that can also be considered an opaque name e.g. /book/chapter[1] What is an opaque name? By opaque name I mean a name that is purely a label. A name that cannot be interpreted as a query expression. Isn't query just another name for name? Sean
Re: file as a directory
Hans Reiser wrote: Sean McGrath wrote: Hans Reiser wrote: Sean McGrath wrote: The thing that interests me most is the difference (if any) between giving a stream of bytes an opaque name e.g. Chapter 1 of my book.sxw versus giving a stream of bytes a query expression that can also be considered an opaque name e.g. /book/chapter[1] What is an opaque name? By opaque name I mean a name that is purely a label. A name that cannot be interpreted as a query expression. Isn't query just another name for name? That is a major philosophical nugget :-) I recommend Saul Kripke's Naming and Necessity: http://www.answers.com/topic/saul-kripke Sean
Re: file as a directory
Sean McGrath wrote: What is an opaque name? By opaque name I mean a name that is purely a label. A name that cannot be interpreted as a query expression. Isn't query just another name for name? That is a major philosophical nugget :-) I recommend Saul Kripke's Naming and Necessity: http://www.answers.com/topic/saul-kripke Sean I suggest considering your opaque names to be what reiser4 calls keys, that is, names that exist for the purpose of finding the object via the storage layer. Hans
