Re: [9fans] Multi-dimensional filesystem
On Fri, Aug 17, 2012 at 09:48:33AM +0200, Lucio De Re wrote: The question here is how to enhance the Unix-like hierarchical directory to embrace multi-dimensionality. I have no doubt that the changes to 9P that will almost certainly be required for this will be fairly obvious and from there building synthetic fileservers will not be difficult. I have a feeling that assuming that Plan 9 already has the required silver bullet is a bad starting point. I don't assume that everything is OK already ;) But I assume what has been (re-)discovered by modern mathematics: that the linearity is fundamental, because it is the way we think. Mathematical language is sequential linear, can speak about multidimensional objects without being itself, lexicaly or syntactically, multidimensional (there have been attempts, by Frege, and even in some early computer languages). So: it is better to be sure that the core, kernel, whatever is agnostic about the dimensions (making no assumptions about, for this, dot-dot), than trying to inject multdimensional features in the core. And one can play with the idea without changing 9P at the moment, just to gather more informations. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
On Thu, Aug 16, 2012 at 04:11:11PM -0400, erik quanstrom wrote: I think you missed the point. What I have given is an example, what indeed made me wonder, initially, about a way to simply store definitions as a text file, the relationships between the notions being described by a directory structure. It was obvious rapidly that this won't do in a classical hierarchical filesystem. i think this can be done with a traditional file system as long as you don't insist that a file belong to exactly one directory. (that is messing with .. is the hard way to go.) (note that plan 9 file servers already do this in a limited way since the dump file system allows an unchanging file or directory to be a member of as many /n/dump//mmdd[.seq] heirarchies as there are dumps between changes.) Yes, the multiplicity of an entry is possible on various systems with various means that are hard or soft links, unions, pointing on shared blocks etc. What is not here is to obtain naturally a view of all the relations of an entry, specially for the parents. The getting dot-dot right is precisely the problem that there are multiple paths, and only one is valid for a classical dir tree and you got to follow this one correct path back. Here, this multiplicity is simultaneously valid, and all the paths are the correct answer. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
The getting dot-dot right is precisely the problem that there are multiple paths, and only one is valid for a classical dir tree and you got to follow this one correct path back. Here, this multiplicity is simultaneously valid, and all the paths are the correct answer. The question here is how to enhance the Unix-like hierarchical directory to embrace multi-dimensionality. I have no doubt that the changes to 9P that will almost certainly be required for this will be fairly obvious and from there building synthetic fileservers will not be difficult. I have a feeling that assuming that Plan 9 already has the required silver bullet is a bad starting point. ++L
Re: [9fans] Multi-dimensional filesystem
What is more bizarre, with my scheme, is how to implement the meaning of ..? If classical clients have to be able to be used, the server must create a fake name (as the penultimate component of the dirpath), that triggers the correct answer from the server. see defmnt.c:/^fixdotdotname for where this is handled by the kernel, not the file server. - erik
Re: [9fans] Multi-dimensional filesystem
and as such, i was thinking of a server that simply distributed requests among a set of servers. so that echo date /net/my-nodes/foo chmod +x /net/my-nodes/foo would work with the normal tools on a normal kernel. all the distribution would be part of a purpose-built fs. How would this work? Someone has to map /net/my-nodes/foo to a set of files. Either a program iterates over the list or you push this list processing into the purpose built filesystem. [Actually the latter is what I was thinking of -- A 9p* protocol would return a list of file handles, and rc* would be an APLish version of rc] i was also thinking of the latter. one could set up a set of parallel trees so that date -n /n/parallel/nodes/dev/date could set the time do some gross level on a set of pre-selected nodes. selection could be fairly obvious, as in echo create notes $machines/n/parallel/ctl. - erik
Re: [9fans] Multi-dimensional filesystem
On Thu, Aug 16, 2012 at 09:40:22AM -0400, erik quanstrom wrote:
What is more bizarre, with my scheme, is how to implement the meaning
of ..? If classical clients have to be able to be used, the server
must create a fake name (as the penultimate component of the
dirpath), that triggers the correct answer from the server.
see defmnt.c:/^fixdotdotname for where this is handled by the kernel,
not the file server.
Well, I find cleanname (libc/port/cleanname.c) doing the .. dance with
compression. But nothing in the kernel proper..
Except by allowing a new syntax .../{a,b,c,d}/foo meaning that foo has
a, b, c and d as parents, the only way I see things working with the
utilities and the .. treatment is to insert a special name ensuring
that a .. suppression leading to this name will trigger the correct
answer from the fileserver.
I don't say that the fileserver has to treat directly the .. in
a path specification: the cleaning is made on the client side before
the 9P request is sent. I mean that for this to work, the fileserver,
from the first mount, will have to create special names (a uniq
name, or the {a,b,c,d} lexeme if the user can have a chance to have
a representation of the graphs that bear some meaning) so that a
request of the content of this special name delivers the correct
view of the data (if classical utilities are used to browse the
file hierarchy).
And by the way, since the parents a, b, c, d can be in several
dimensions, and not having the same scalar value as a coordinate
(`a' being direct child of /; `b' being deep in multiple directions
etc., going up is not going up 1 in every dimension...), going to
the parents would not mean anything, and the best to do would be
to present a fake middle node where {a, b, c, d} appear in .-/
(parents), the children of {a,b,c,d} in .+/, and nothing in ./
(user to decide precisely where in a named node, he wants to go,
standing in between for the moment).
In a more general term, as projective geometry gives rules to
represent a 3D objects on a 2D plane, is it possible to represent
multiple dimensions in a single dimension string? The answer is
yes, since this is what the n-uples coordinates representation is and
since our linear languages speak about multidimensional features even
when we have strictly no intuition about these spaces.
Can this be user friendly? No, if there are a lot of links, and
long enumerations. But the naming of the nodes is always a problem
(lengthy names, spaces, discrimining characters being put last and
not first bringing identical long prefixes etc.).
--
Thierry Laronde tlaronde +AT+ polynum +dot+ com
http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
On Thu, 16 Aug 2012 09:40:22 EDT erik quanstrom quans...@quanstro.net wrote:
What is more bizarre, with my scheme, is how to implement the meaning
of ..? If classical clients have to be able to be used, the server
must create a fake name (as the penultimate component of the
dirpath), that triggers the correct answer from the server.
see defmnt.c:/^fixdotdotname for where this is handled by the kernel,
not the file server.
It pretty much has to. Consider what happens when you do
something like
% x=`{pwd}
% bind /sys/src tmp
% cd tmp
% cd ..
This gets you back to $x. If you leave .. upto the
fileserver, you'd get back to /sys not $x. The server can't
know the right context.
Re: [9fans] Multi-dimensional filesystem
On Thu Aug 16 11:44:01 EDT 2012, tlaro...@polynum.com wrote:
On Thu, Aug 16, 2012 at 09:40:22AM -0400, erik quanstrom wrote:
What is more bizarre, with my scheme, is how to implement the meaning
of ..? If classical clients have to be able to be used, the server
must create a fake name (as the penultimate component of the
dirpath), that triggers the correct answer from the server.
see defmnt.c:/^fixdotdotname for where this is handled by the kernel,
not the file server.
Well, I find cleanname (libc/port/cleanname.c) doing the .. dance with
compression. But nothing in the kernel proper..
cleanname is called by the function i indicated in devmnt.
Except by allowing a new syntax .../{a,b,c,d}/foo meaning that foo has
a, b, c and d as parents, the only way I see things working with the
utilities and the .. treatment is to insert a special name ensuring
that a .. suppression leading to this name will trigger the correct
answer from the fileserver.
this is already the case due to union directories (as bakul points out),
and the solution has been widely discussed.
- erik
Re: [9fans] Multi-dimensional filesystem
On Thu, Aug 16, 2012 at 12:06:40PM -0400, erik quanstrom wrote:
Except by allowing a new syntax .../{a,b,c,d}/foo meaning that foo has
a, b, c and d as parents, the only way I see things working with the
utilities and the .. treatment is to insert a special name ensuring
that a .. suppression leading to this name will trigger the correct
answer from the fileserver.
this is already the case due to union directories (as bakul points out),
and the solution has been widely discussed.
I gather. But here, there is an union on request (descending), and a
split once you go up, since it is not the purpose to hide the
explication of the union...
--
Thierry Laronde tlaronde +AT+ polynum +dot+ com
http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
On Thu, Aug 16, 2012 at 08:59:01AM -0700, Bakul Shah wrote:
On Thu, 16 Aug 2012 09:40:22 EDT erik quanstrom quans...@quanstro.net
wrote:
What is more bizarre, with my scheme, is how to implement the meaning
of ..? If classical clients have to be able to be used, the server
must create a fake name (as the penultimate component of the
dirpath), that triggers the correct answer from the server.
see defmnt.c:/^fixdotdotname for where this is handled by the kernel,
not the file server.
It pretty much has to. Consider what happens when you do
something like
% x=`{pwd}
% bind /sys/src tmp
% cd tmp
% cd ..
This gets you back to $x. If you leave .. upto the
fileserver, you'd get back to /sys not $x. The server can't
know the right context.
And this is why the pathname has to encode, one way or the other, the
hierarchy, to let the fileserver be context ignorant, and the user
go wherever he wants.
--
Thierry Laronde tlaronde +AT+ polynum +dot+ com
http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
It pretty much has to. Consider what happens when you do something like
% x=`{pwd}
% bind /sys/src tmp
% cd tmp
% cd ..
This gets you back to $x. If you leave .. upto the fileserver, you'd get
back to /sys not $x. The server can't know the right context.
I thought this problem was solved in plan9:
Lexical File Names in Plan 9, or, Getting Dot-Dot Right - Rob Pike
A vexing old problem solved: how to make pwd get the right answer in the face
of multiply-bound directories.
http://plan9.bell-labs.com/sys/doc/lexnames.html
--
Burton Samograd
This e-mail, including accompanying communications and attachments, is strictly
confidential and only for the intended recipient. Any retention, use or
disclosure not expressly authorised by Markit is prohibited. This email is
subject to all waivers and other terms at the following link:
http://www.markit.com/en/about/legal/email-disclaimer.page
Please visit http://www.markit.com/en/about/contact/contact-us.page? for
contact information on our offices worldwide.
Re: [9fans] Multi-dimensional filesystem
First one, related to what I was wandering about, is mathematical definitions and relationships. Take the picture of the first volume of van der Waerden's Albebra (I have the german edition and will keep the german words). We speak about links between notions presented in a linear order: the order of the chapters: Galois theory chapter is the descendant of concepts from Vektorräume, Fortsetzung der Gruppentheorie and Körpertheorie chapters; Vektorräume and Gruppentheory being siblings (children of Ringe und Körper) while Körpertheorie is a grand-child of Vektorräume. Example of multiple parents, and not at the same level. This seems to be possible to recreate with befs, but I have to say: this particular problem is too specific to be solved with something like a filesystem. One should write a fileserver only when that would allow the user to use multiple familiar tools to work on the task. With your example it looks like the only thing the user would need here is the list of related topics along with the description of their relationship so that a related topic could be chosen and read. You can use a relational database for storage of the content along with the list of related topics and a program that can browse that database and follow the links. A user would never even think about doing something like 'cp ./Galors theory/parents ./Galors theory/children, even though the approach of implementing that in a fileserver allows that.
Re: [9fans] Multi-dimensional filesystem
On Thu, Aug 16, 2012 at 08:02:52PM +0300, Eugene Gorodinsky wrote: First one, related to what I was wandering about, is mathematical definitions and relationships. Take the picture of the first volume of van der Waerden's Albebra (I have the german edition and will keep the german words). We speak about links between notions presented in a linear order: the order of the chapters: Galois theory chapter is the descendant of concepts from Vektorräume, Fortsetzung der Gruppentheorie and Körpertheorie chapters; Vektorräume and Gruppentheory being siblings (children of Ringe und Körper) while Körpertheorie is a grand-child of Vektorräume. Example of multiple parents, and not at the same level. This seems to be possible to recreate with befs, but I have to say: this particular problem is too specific to be solved with something like a filesystem. One should write a fileserver only when that would allow the user to use multiple familiar tools to work on the task. With your example it looks like the only thing the user would need here is the list of related topics along with the description of their relationship so that a related topic could be chosen and read. You can use a relational database for storage of the content along with the list of related topics and a program that can browse that database and follow the links. A user would never even think about doing something like 'cp ./Galors theory/parents ./Galors theory/children, even though the approach of implementing that in a fileserver allows that. I think you missed the point. What I have given is an example, what indeed made me wonder, initially, about a way to simply store definitions as a text file, the relationships between the notions being described by a directory structure. It was obvious rapidly that this won't do in a classical hierarchical filesystem. Say it is a thought experiment. I don't plan to write a fileserver just to implement this example. But multidimensional data is more widely needed than this example. I have indeed cases where a fileserver like that would allow storage of some data already stored in a database (served by a database SQL engine, for example), and for which the classical relational database is simply not a match. Files have the benefits that you can use the text tools on them to grep or whatever. And the file hierarchy reflects what is given by indexing in a relational database. Last, I always find it useful when I program or when I learn to ask me questions about what if (as long as I don't spend the bulk of time beating around the bush). One learns with exercices, questions, reading a book a pen in the hand, simply because this is the only way to leave a track in one's mind. Even dead ends are useful, since you know what is possible and what is not. And I may do strictly nothing, at the moment, with some multidimensional stuff, but use some of the solutions sketched, or something thought about passing, or part of a detail, to solve a non directly related problem later. There is a paper by D.E. Knuth, IIRC, speaking about the usefulness of toy programming. This is it. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
I think you missed the point. What I have given is an example, what indeed made me wonder, initially, about a way to simply store definitions as a text file, the relationships between the notions being described by a directory structure. It was obvious rapidly that this won't do in a classical hierarchical filesystem. i think this can be done with a traditional file system as long as you don't insist that a file belong to exactly one directory. (that is messing with .. is the hard way to go.) (note that plan 9 file servers already do this in a limited way since the dump file system allows an unchanging file or directory to be a member of as many /n/dump//mmdd[.seq] heirarchies as there are dumps between changes.) - erik
Re: [9fans] Multi-dimensional filesystem
I'm sure I must not understand the problem fully and am confused because of that, but how is this idea of multidimensionality different from a relational filesystem approach such as befs ( http://www.nobius.org/~dbg/practical-file-system-design.pdf)? 2012/8/5 tlaro...@polynum.com On Sun, Aug 05, 2012 at 11:29:19AM -0400, Charles Forsyth wrote: There is also http://lsub.org/iwp9/cready/abheyshahplan9.pdf Thank you! Indeed, this is related. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
On Wed, Aug 15, 2012 at 07:04:28PM +0300, Eugene Gorodinsky wrote: I'm sure I must not understand the problem fully and am confused because of that, but how is this idea of multidimensionality different from a relational filesystem approach such as befs ( http://www.nobius.org/~dbg/practical-file-system-design.pdf)? One can obviously mimick what I have described with a database, fields, the relationships being built by indexing. The hierarchical (but with multiple parents as well as multiple children) is a representation of the indexing in this case. So the BeFS could---from a cursory look about the combination of database, indexing and hierachy representation---offer a solution, except that the multiple parents, is not there (this could be, as well as with other systems, be done with .+ and .- presentation). The main differences from what I have in mind are: 1) There is no general relational database concept: the relationships of the records (files, that can have both a text content [the definition in my example] and be a directory node) is exclusively isomorphic to coordinates: (i, j, k, ...). 2) There is no constraint in the size of the fields: the dimension can grow with time (no given dimensions coordinates being, by convention, equal to 0 to reach the actual dimension) ; there is no limit (except implementation one) for the size of an enumeration. 3) The hierarchy is the user interface, to view and to add the data: a new file (record) is added by placing it in the hierarchy; while in a relational database, the indexing is deduced from the actual records; here, so to say, the data is entered through the indexing. 4) And the problem was also thought through 9P: is there something in 9P that would prevent, at least theoritically, such a view of data to be presented? With the convention of .+ and .-, my answer is no: 9P has no hardcoded knowledge of .. if I'm not mistaken. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
On Wed, 15 Aug 2012 19:33:27 +0200 tlaro...@polynum.com wrote: The main differences from what I have in mind are: 1) There is no general relational database concept: the relationships of the records (files, that can have both a text content [the definition in my example] and be a directory node) is exclusively isomorphic to coordinates: (i, j, k, ...). Sounds like you want to represent a node with a directory, that has a content file that stores content associated with this node and a bunch of links to other nodes. You are most interested in parent/child relationships so for instance you can store links to all parents in a parent file and links to all children in children file. Not suggesting this is how you implement it; I am just trying to understand. A concrete example from you would help. 2) There is no constraint in the size of the fields: the dimension can grow with time (no given dimensions coordinates being, by convention, equal to 0 to reach the actual dimension) ; there is no limit (except implementation one) for the size of an enumeration. 3) The hierarchy is the user interface, to view and to add the data: a new file (record) is added by placing it in the hierarchy; while in a relational database, the indexing is deduced from the actual records; here, so to say, the data is entered through the indexing. 4) And the problem was also thought through 9P: is there something in 9P that would prevent, at least theoritically, such a view of data to be presented? With the convention of .+ and .-, my answer is no: 9P has no hardcoded knowledge of .. if I'm not mistaken. Note that in Unix .. came for free since each dir stored the inode number (a link to an anonymous inode) along with a name. The scheme could represent any graph but the kernel restricted it to a tree by disallowing links to directories. Expanding this back to multiple parents in effect makes a poset (partially ordered set) but the unix trick or storing (parent-inode, ..) fails. Treating .. as an /operator/ that cancels the previous component in a path also fails. You can perhaps extend a symlink object to store multiple links as I described above. The other difficulty is that something like 9p will only yield zero or one object as you traverse a path. How do you handle multiple parents? A more general idea is to consider a path component a graph expression component that is interpreted by the current node and may yield *any number of* new nodes. In other words x = open(a/b/c, mode) can yield a vector of file descriptors! Leaving component interpretation to the current node makes this a very dynamic and powerful system (for example, one can think of a node that maps to a list of network nodes -- so something like echo date /net/my-nodes/foo chmod +x /net/my-nodes/foo /net/my-nodes/foo can run date on all my nodes! This would be a very compact expression of how to distribute data or work. Your original query is vague enough ( confusing, at least to me) that one can take it in a number of directions :-)
Re: [9fans] Multi-dimensional filesystem
x = open(a/b/c, mode) can yield a vector of file descriptors! Leaving component interpretation to the current node makes this a very dynamic and powerful system (for example, one can think of a node that maps to a list of network nodes -- so something like echo date /net/my-nodes/foo chmod +x /net/my-nodes/foo /net/my-nodes/foo one would think that a distributing file server would be a better abstraction as the normal tools could be brought to bear on the problem. - erik
Re: [9fans] Multi-dimensional filesystem
On Wed, 15 Aug 2012 16:17:28 EDT erik quanstrom quans...@labs.coraid.com wrote: x = open(a/b/c, mode) can yield a vector of file descriptors! Leaving component interpretation to the current node makes this a very dynamic and powerful system (for example, one can think of a node that maps to a list of network nodes -- so something like echo date /net/my-nodes/foo chmod +x /net/my-nodes/foo /net/my-nodes/foo one would think that a distributing file server would be a better abstraction as the normal tools could be brought to bear on the problem. I was just exploring the APLish nature of the idea. I used shell syntax to get the idea across -- here my-nodes would map to a set of fileserver nodes that interpret the remaining path so this path actually maps to a whole bunch of files. No idea if this is better, worse or even sensible. There are some graph languages that could possibly map here. What is a distributing file server? If you mean something like cxfs, luster, glusterfs, ceph, etc they wouldn't do the same thing. The above would in fact be kind of a distributed filesystem!
Re: [9fans] Multi-dimensional filesystem
On Wed, Aug 15, 2012 at 01:09:49PM -0700, Bakul Shah wrote: Sounds like you want to represent a node with a directory, that has a content file that stores content associated with this node and a bunch of links to other nodes. You are most interested in parent/child relationships so for instance you can store links to all parents in a parent file and links to all children in children file. Not suggesting this is how you implement it; I am just trying to understand. A concrete example from you would help. I will give two (DeLuxe(R) version!) examples. First one, related to what I was wandering about, is mathematical definitions and relationships. Take the picture of the first volume of van der Waerden's Albebra (I have the german edition and will keep the german words). We speak about links between notions presented in a linear order: the order of the chapters: Galois theory chapter is the descendant of concepts from Vektorräume, Fortsetzung der Gruppentheorie and Körpertheorie chapters; Vektorräume and Gruppentheory being siblings (children of Ringe und Körper) while Körpertheorie is a grand-child of Vektorräume. Example of multiple parents, and not at the same level. Second example: field parcel (surveyor work). One can divide a parcel, leading to several children. But one can also reunite several adjacent parcels leading to one uniq new, all the inner borders being suppressed. This new parcel has several parents. And the child is bigger than every parent taken alone... 4) And the problem was also thought through 9P: is there something in 9P that would prevent, at least theoritically, such a view of data to be presented? With the convention of .+ and .-, my answer is no: 9P has no hardcoded knowledge of .. if I'm not mistaken. Note that in Unix .. came for free since each dir stored the inode number (a link to an anonymous inode) along with a name. The scheme could represent any graph but the kernel restricted it to a tree by disallowing links to directories. Expanding this back to multiple parents in effect makes a poset (partially ordered set) but the unix trick or storing (parent-inode, ..) fails. Treating .. as an /operator/ that cancels the previous component in a path also fails. Yes. You can perhaps extend a symlink object to store multiple links as I described above. The other difficulty is that something like 9p will only yield zero or one object as you traverse a path. How do you handle multiple parents? A more general idea is to consider a path component a graph expression component that is interpreted by the current node and may yield *any number of* new nodes. In other words x = open(a/b/c, mode) can yield a vector of file descriptors! Leaving component interpretation to the current node makes this a very dynamic and powerful system (for example, one can think of a node that maps to a list of network nodes -- so something like echo date /net/my-nodes/foo chmod +x /net/my-nodes/foo /net/my-nodes/foo can run date on all my nodes! This would be a very compact expression of how to distribute data or work. Your original query is vague enough ( confusing, at least to me) that one can take it in a number of directions :-) I don't claim that my question was definite and it was open. So every idea wandering around this fuzzy concept is OK :-) But in my idea, there would be several distinct things (and I think plan9 has already split the things making alternative view possible): 1) There is the storage of the things. This is mostly orthogonal---the trick being, as always, to be able to retrieve efficiently the stuff but really starting from the coordinates (i, j, k, ...). 2) In 9P, the .. if I'm not mistaken is not here. One can request a file, or go back to the previous one. The .. including path removing is not implemented in 9P if I read correctly the manpage. So a client can speak 9P, but the way the stuff are presented will be mainly in the server, the trick to allow existing clients to browse being this .+ and .-: .+ leads to children; .- leads to parents and a DESC file being perhaps a text description, and some fake parenting allowing a request of .. to trigger a special action from the server---but all these are artefacts of the server. 3) This is the server that offers a view of the data. And in this case, indeed, a request for something can trigger bizarre actions. From a very cursory look, it seems to me that it could be simpler to implement this with Plan9, because it is relatively agnostic about what is a filesystem, than with an Unix if I take into account the difficulties Unix kernels have with userland filesystems---probably because a lot of semantics are assumed in the vnode subsystem. By conception, Plan9 makes a distinction between viewing the data (terminal), processing the data (CPU) and serving the data (fileserver). This is the case here, and this is not a surprise since
Re: [9fans] Multi-dimensional filesystem
I was just exploring the APLish nature of the idea. I used shell syntax to get the idea across -- here my-nodes would map to a set of fileserver nodes that interpret the remaining path so this path actually maps to a whole bunch of files. No idea if this is better, worse or even sensible. There are some graph languages that could possibly map here. What is a distributing file server? If you mean something like cxfs, luster, glusterfs, ceph, etc they wouldn't do the same thing. The above would in fact be kind of a distributed filesystem! i was thinking of file server in the traditional (ahem) plan 9 sense, a network service that responds to 9p rather than a traditional (boring) unix-style store. and as such, i was thinking of a server that simply distributed requests among a set of servers. so that echo date /net/my-nodes/foo chmod +x /net/my-nodes/foo would work with the normal tools on a normal kernel. all the distribution would be part of a purpose-built fs. - erik
Re: [9fans] Multi-dimensional filesystem
On Wed, 15 Aug 2012 23:27:34 +0200 tlaro...@polynum.com wrote: I will give two (DeLuxe(R) version!) examples. Thanks. Sounds like you are taking about IS-A and HAS-A relationships. 2) In 9P, the .. if I'm not mistaken is not here. One can request a file, or go back to the previous one. The .. including path removing is not implemented in 9P if I read correctly the manpage. So a client can speak 9P, but the way the stuff are presented will be mainly in the server, the trick to allow existing clients to browse being this .+ and .-: .+ leads to children; .- leads to parents and a DESC file being perhaps a text description, and some fake parenting allowing a request of .. to trigger a special action from the server---but all these are artefacts of the server. Looks like you are treating .+ .- specially *somewhere*. Are foo/.- and foo/bar treated differently? What would foo/.- even mean? What would open(foo/.-, mode) do? When I try to work out an example I don't get how you can implement this. May be you can use your special syntax to present an example or two as shell scripts? In any case since multiple relationships are possible, why single out a specific case? That is, it seems to me you can either use the current 9p as is and implement what you want by convention in your programs or extend the model in a major way (which is where I was going). From a very cursory look, it seems to me that it could be simpler to implement this with Plan9, because it is relatively agnostic about what is a filesystem, than with an Unix if I take into account the difficulties Unix kernels have with userland filesystems---probably because a lot of semantics are assumed in the vnode subsystem. You can use fusefs to implement a plan9 like synthetic filesystems. But compared to plan9 the unix model is indeed far more constrained and a lot more complicated -- you should see Linux's /sys synthetic filesystem -- it is also a good exmaple of what multiple inheritance wreak.
Re: [9fans] Multi-dimensional filesystem
On Wed, 15 Aug 2012 21:38:43 EDT erik quanstrom quans...@quanstro.net wrote: i was thinking of file server in the traditional (ahem) plan 9 sense, a network service that responds to 9p rather than a traditional (boring) unix-style store. To me the plan9 model is the boring one. Boringly simple. And I like it that way! Regardless, both can be used to distribute files. and as such, i was thinking of a server that simply distributed requests among a set of servers. so that echo date /net/my-nodes/foo chmod +x /net/my-nodes/foo would work with the normal tools on a normal kernel. all the distribution would be part of a purpose-built fs. How would this work? Someone has to map /net/my-nodes/foo to a set of files. Either a program iterates over the list or you push this list processing into the purpose built filesystem. [Actually the latter is what I was thinking of -- A 9p* protocol would return a list of file handles, and rc* would be an APLish version of rc]
Re: [9fans] Multi-dimensional filesystem
On Wed, Aug 15, 2012 at 08:47:47PM -0700, Bakul Shah wrote: 2) In 9P, the .. if I'm not mistaken is not here. One can request a file, or go back to the previous one. The .. including path removing is not implemented in 9P if I read correctly the manpage. So a client can speak 9P, but the way the stuff are presented will be mainly in the server, the trick to allow existing clients to browse being this .+ and .-: .+ leads to children; .- leads to parents and a DESC file being perhaps a text description, and some fake parenting allowing a request of .. to trigger a special action from the server---but all these are artefacts of the server. Looks like you are treating .+ .- specially *somewhere*. Are foo/.- and foo/bar treated differently? What would foo/.- even mean? What would open(foo/.-, mode) do? They are treated specially by the server serving 9P requests. The aim would be to present a classical file hierarchy, even with special conventional names, implementing multiple parents, by convention, under .-/, and (classical) multiple children by .+/. .+/ holds the children (if the requested node has ones). .-/ holds the parents (in reverse order i.e. the nearest parents up in every dimension). ./* being the siblings of the node (siblings, whether files or directories appear by name not hidden under .+/ or .-/). What is more bizarre, with my scheme, is how to implement the meaning of ..? If classical clients have to be able to be used, the server must create a fake name (as the penultimate component of the dirpath), that triggers the correct answer from the server. But the result is that going up means going down (from the view served) to the next hierarchy level in .-/. When I try to work out an example I don't get how you can implement this. May be you can use your special syntax to present an example or two as shell scripts? In any case since multiple relationships are possible, why single out a specific case? That is, it seems to me you can either use the current 9p as is and implement what you want by convention in your programs or extend the model in a major way (which is where I was going). The two approaches are not mutually exclusive. I mean one can implement using the current 9p (to see if it works, what are the limits, and the benefits...); and later tackle a more refined and perhaps more general answer the way you are describing. Plus, for a more broader view, I am---as I think a lot of users/programmers---not absolutely happy with relational databases. For example, for KerGIS, I have implemented, for attributes linked to geometry, a library, with a SQL backend (for use with PostgreSQL), a DBF (mainly for ESRI(R) Shape support), and a stdout and a stdin to be able to have a text format and simply pipe to utilities to save, fill or convert between databases. Mostly, PostgreSQL for a normal use (of KerGIS) is overkill. And the SQL type does not allow the kind of relationships I want---multiple parents are difficult, or rely on processing the data for some indexing; to have a list in one field, without specifying a length is not there etc. Having a file system organizing the relationships between the data, and implementing, one can say, the indexing by its very structure, and allowing to retrieve records or fields by the file system would be ideal (a small loss of time is not a problem). Not to say that once ownership, locking etc. is made by a fileserver, you have a distributed system for free---a lot of GIS softwares are using relational database even to store geometry! (a disaster from an efficiency standpoint when one is making geometrical manipulations---processing spaghettis and so on), simply because this relieves the developers/programmers from the distributed problems: they let the database server handles this; IMHO if databases are so pervasive nowadays, this is because of this. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
There is also http://lsub.org/iwp9/cready/abheyshahplan9.pdf
Re: [9fans] Multi-dimensional filesystem
On Sun, Aug 05, 2012 at 11:29:19AM -0400, Charles Forsyth wrote: There is also http://lsub.org/iwp9/cready/abheyshahplan9.pdf Thank you! Indeed, this is related. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
Hello, On Fri, Aug 03, 2012 at 05:08:52PM -0400, Burton Samograd wrote: Has someone ever played with the notion of a multidimensional filesystem David Korn did some research on a 3d file system called 3d: David G. Korn, Eduardo Krell, The 3-D File System, pp147-156, USENIX Conference Proceedings, Summer 1989, Baltimore, MD And also at behind a paywall: http://onlinelibrary.wiley.com/doi/10.1002/spe.4380201304/abstract I seem to remember it being available through ast: http://www2.research.att.com/sw/download/ Thank you for the reference! Best, -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
I'm pretty sure there's an isomorphism between your multidimensional filesystem and the set of potential namespaces. A dimension in your multidimensional fileysystem is just an arbitrary set of filesystems bounded in a particular namespace... -- Aram Hăvărneanu
Re: [9fans] Multi-dimensional filesystem
On 03/08/2012 19:18, tlaro...@polynum.com wrote:
Hello,
This is mainly a theoretical question.
While playing with the representation of mathematical definitions as
a file hierarchy (at dot you find a DESC or whatever named file with
the description, and the subdirs are simply more restrictive
instances of the thing; say : collection - magma - monoïde -
group etc.), it is soon obvious that a filesystem is a one dimension
thing: you only follow one string. Multiple parents at the same
level are not there.
One could trick partly using hard or soft links. But with always the
same problem: who is dot-dot, in a case where multiple parents are
here? And multiple parents are not, to my knowledge, supported by
kernel filesystem code. Manipulating the namespace is not the same.
Has someone ever played with the notion of a multidimensional
filesystem, where '/' is the origin, the nodes would be some
representation of (a, b, c,...) (even negatives perhaps), each node
having a name (user defined one by the way), and if a node is, say
(3, 0, 1,...) this means that it is to be found as the third subdir
of the (1, 0, 0,...) path etc., (In this scheme, if there is no link
(no path) from another notion, it is another dimension).
Just for intellectual curiosity.
Best,
Hi,
As far as I understand, it seems you are interested in the idea of views
over your files. Something that has been approached as non-hierarchical
file systems. But the complexity of handling such graphs often seems to
be the reason why these projects failed. Not to mention how the pain to
adapt them to the existing systems that are strictly hierarchical.
In a project we presented in iwp9 6e (2011), we introduced the design of
a filesystem that stores files (stream or hierarchy of files) in
/records/ on a WORM file system (Venti). Records are stored in sequence,
as they arrive (in respect to their recording order). Each record is
identified by your username 'u', the repository name 'h' (for host) and
an index related to time 't' (that is not time, but a self incremented
interger linked to the date in the current calendar). Each triplet
{u,h,t} is uniquely linked to a Venti SHA-1 score. That is the ground level.
On top of that, you are free to point files to build any view on them
and store these views again in new records, using any naming convention
you like at this moment. A non intrusive hypertext language can be
used to write a log book from were you point the files you stored in
records. From this log book, you can do literate programming, run
scripts on files, etc. (For the moment, Emacs org-mode seems to be a
good hyper-text language to start from.)
A third component, a triplet indexer, helps you to find which records
points to other records, an vice versa. A plain text search engine
helps you to retrieve text from the past. All is centered on traceability.
We spent a lot of time studying reference bibliography from the 60s to
today and already have plans for implementation of the ground level
file system on top of Venti.
Hope this will interest you, at least just for intellectual curiosity!
;-)
Nicolas
Re: [9fans] Multi-dimensional filesystem
On Sat, Aug 04, 2012 at 01:56:07PM +0300, Aram H?v?rneanu wrote: I'm pretty sure there's an isomorphism between your multidimensional filesystem and the set of potential namespaces. A dimension in your multidimensional fileysystem is just an arbitrary set of filesystems bounded in a particular namespace... I'm not quite sure about the bijective property but at least, since mathematics has recognized that the linear relationship is fundamental, and since base and dimensions are easily approached by linear Algebra, it seems probable enough that as long as the dot-dot thing is implemented by a fileserver---no semantics enforced at the kernel level---once you have the primitive for an oriented linearity (children/parent), multidimension is achievable. As sketched in a previous mail, a fileserver could present a classical file hierarchy, but with ., .., and, say .+ for the hierarchical children, and .- for the reverse hierarchical parents (these being only a _view_; storage is another story...). So this could be achieved but by not attempting to provide an ubicuitous view, but only a local view from the file considered. -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
On Sat, Aug 04, 2012 at 02:16:33PM +0200, Nicolas Bercher wrote:
As far as I understand, it seems you are interested in the idea of views
over your files. Something that has been approached as non-hierarchical
file systems. But the complexity of handling such graphs often seems to
be the reason why these projects failed. Not to mention how the pain to
adapt them to the existing systems that are strictly hierarchical.
It seems clearer (and this is in accordance with notions expressed
further in your message), that the file hierarchy is a view of the data.
The way the data is stored can be totally different from the classical
one dimension tree structure.
The multidimension can (seen from far) be tricked with (as explained in
another message), from the current file, a ., .., .+ for children
and .- for parents; keeping in mind that the paths are only the next
neighbours (so this is a limited local view), as if say in a 2D grid
(discrete coordinates) you will only consider the next objects with
coordinates differing only by +/- 1.
In a project we presented in iwp9 6e (2011), we introduced the design of
a filesystem that stores files (stream or hierarchy of files) in
/records/ on a WORM file system (Venti). Records are stored in sequence,
as they arrive (in respect to their recording order). Each record is
identified by your username 'u', the repository name 'h' (for host) and
an index related to time 't' (that is not time, but a self incremented
interger linked to the date in the current calendar). Each triplet
{u,h,t} is uniquely linked to a Venti SHA-1 score. That is the ground level.
On top of that, you are free to point files to build any view on them
and store these views again in new records, using any naming convention
you like at this moment. A non intrusive hypertext language can be
used to write a log book from were you point the files you stored in
records. From this log book, you can do literate programming, run
scripts on files, etc. (For the moment, Emacs org-mode seems to be a
good hyper-text language to start from.)
A third component, a triplet indexer, helps you to find which records
points to other records, an vice versa. A plain text search engine
helps you to retrieve text from the past. All is centered on traceability.
We spent a lot of time studying reference bibliography from the 60s to
today and already have plans for implementation of the ground level
file system on top of Venti.
My application case was a little different. This was to present a
database of mathematical definitions with a file hierarchy, the
relationship between the notions being represented by the file hierarchy
(but this obviously needs multidimension since not everything is on the
same deducing string of notions).
In this case, there will be some lookup function, taking a name or a
more complex criteria, and returning a file with the definition
represented by a n-uple (a, b, c, ...). The advantage is that the
identifier can be of fixed length or by convention, if one more dimension is
added, it is added last, and shorter uples have all zeroes
coordinates for non expressed dimensions. But this encodes not only the
definition search, but all the relationships since parent notions or
descendant notions are immediately deduced from this.
So there could be a WORM but with a typical copy-on-write block strategy
for backup and history (but this is orthogonal to the multidimension).
The implementation of the multidimension per se (as soon as there is
a matrix representation, there is an obvious way to do). And the
representation (view) of the data as a normal file structure with
the .+ and .-.
(This explanation does not hide the vagueness of the thing---but I'm
just wandering intellectually at the moment).
Hope this will interest you, at least just for intellectual curiosity!
;-)
Yes ;-)
--
Thierry Laronde tlaronde +AT+ polynum +dot+ com
http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
if i understand correctly, this is one way it could be done (i think): * built a graph representing the structure * create a file server that given a graph and a root node, synthesizes a hierarchy, AND * on every walk to a node launches a copy of itself with the same graph but the new node as the root AND * mounts the newly launched copy of it self under that node (like exportfs). -Skip On Fri, Aug 3, 2012 at 10:18 AM, tlaro...@polynum.com wrote: Hello, This is mainly a theoretical question. While playing with the representation of mathematical definitions as a file hierarchy (at dot you find a DESC or whatever named file with the description, and the subdirs are simply more restrictive instances of the thing; say : collection - magma - monoïde - group etc.), it is soon obvious that a filesystem is a one dimension thing: you only follow one string. Multiple parents at the same level are not there. One could trick partly using hard or soft links. But with always the same problem: who is dot-dot, in a case where multiple parents are here? And multiple parents are not, to my knowledge, supported by kernel filesystem code. Manipulating the namespace is not the same. Has someone ever played with the notion of a multidimensional filesystem, where '/' is the origin, the nodes would be some representation of (a, b, c,...) (even negatives perhaps), each node having a name (user defined one by the way), and if a node is, say (3, 0, 1,...) this means that it is to be found as the third subdir of the (1, 0, 0,...) path etc., (In this scheme, if there is no link (no path) from another notion, it is another dimension). Just for intellectual curiosity. Best, -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
On Fri, Aug 03, 2012 at 11:58:08AM -0700, Skip Tavakkolian wrote: if i understand correctly, this is one way it could be done (i think): * built a graph representing the structure * create a file server that given a graph and a root node, synthesizes a hierarchy, AND * on every walk to a node launches a copy of itself with the same graph but the new node as the root AND * mounts the newly launched copy of it self under that node (like exportfs). Interesting. I guess the children/parents problem could be tricked by this translation (changing the origin on each node, and in fact presenting parents as one subdirs tree [reverse hierarchy] and the real children classically the other part (the negative could be precisely parents)). But in this case the equivalent of cd .. on root would indeed goes in a subdir, unless root is absolute root... What is typical is that n 9P, walk takes whether a subdir or the previous, that is the .. is really a local variable meaning whatever was before and not something hard encoded in the current file. The problem of implementing something efficient for the storage (not serving it) is another matter. The lack of links (whether hard or symbolic) on Plan9 could seem to suppress some facilities. But since this does not give all, even not a lot (this does not address multidimensional), it happens that it could be easier to implement something like this in a Plan9 world... Thanks! -- Thierry Laronde tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
Re: [9fans] Multi-dimensional filesystem
Has someone ever played with the notion of a multidimensional filesystem David Korn did some research on a 3d file system called 3d: David G. Korn, Eduardo Krell, The 3-D File System, pp147-156, USENIX Conference Proceedings, Summer 1989, Baltimore, MD And also at behind a paywall: http://onlinelibrary.wiley.com/doi/10.1002/spe.4380201304/abstract I seem to remember it being available through ast: http://www2.research.att.com/sw/download/ -- Burton Samograd This e-mail, including accompanying communications and attachments, is strictly confidential and only for the intended recipient. Any retention, use or disclosure not expressly authorised by Markit is prohibited. This email is subject to all waivers and other terms at the following link: http://www.markit.com/en/about/legal/email-disclaimer.page Please visit http://www.markit.com/en/about/contact/contact-us.page? for contact information on our offices worldwide.
Re: [9fans] Multi-dimensional filesystem
On Fri, Aug 03, 2012 at 05:08:52PM -0400, Burton Samograd wrote: This e-mail, including accompanying communications and attachments, is strictly confidential and only for the intended recipient. Any retention, use or disclosure not expressly authorised by Markit is prohibited. This email is subject to all waivers and other terms at the following link: http://www.markit.com/en/about/legal/email-disclaimer.page Please visit http://www.markit.com/en/about/contact/contact-us.page? for contact information on our offices worldwide. Are you kidding
Re: [9fans] Multi-dimensional filesystem
It's automatically added to my mails. Sorry, I forget about it because I don't add it, the mail server does. -Original Message- From: 9fans-boun...@9fans.net [mailto:9fans-boun...@9fans.net] On Behalf Of Kurt H Maier Sent: Friday, August 03, 2012 3:13 PM To: Fans of the OS Plan 9 from Bell Labs Subject: Re: [9fans] Multi-dimensional filesystem On Fri, Aug 03, 2012 at 05:08:52PM -0400, Burton Samograd wrote: This e-mail, including accompanying communications and attachments, is strictly confidential and only for the intended recipient. Any retention, use or disclosure not expressly authorised by Markit is prohibited. This email is subject to all waivers and other terms at the following link: http://www.markit.com/en/about/legal/email-disclaimer.page Please visit http://www.markit.com/en/about/contact/contact-us.page? for contact information on our offices worldwide. Are you kidding This e-mail, including accompanying communications and attachments, is strictly confidential and only for the intended recipient. Any retention, use or disclosure not expressly authorised by Markit is prohibited. This email is subject to all waivers and other terms at the following link: http://www.markit.com/en/about/legal/email-disclaimer.page Please visit http://www.markit.com/en/about/contact/contact-us.page? for contact information on our offices worldwide.
