Re: [9fans] wstat and atomic directory change
option 2 doesn't make sense. using the cpu(1) as a test setup, please show how such a wstat would work atomically if one does 'mv /mnt/term/foo /tmp/foo'. note that 'cpu' exports the local fs to the cpu it connects to (which is then mounted on /mnt/term). also one can 'cpu' to a different domain (i.e different fs). On Thu Feb 05 2015 at 8:47:43 AM Giacomo Tesio wrote: > Il 05/Feb/2015 17:26 "erik quanstrom" ha scritto: > > > > I'm wondering for a > > > synthetic filesystem in which, when you move a folder in a special > > > directory, something magic happens. > > > As far as I can see, it is not possible with a 9p2000 fileservice, is > it? > > > > i don't see why you can't make a magic directory that works that way. > > > > How? Which 9p message would trigger the magic? > > I just see 2 options: > 1. Have a control file > 2. Use custom wstat with full path to move > > But if both are wrong, I'm lost. > > Giacomo >
Re: [9fans] wstat and atomic directory change
On Thu, 05 Feb 2015 08:20:30 PST erik quanstrom wrote: > > All this reflections arise from the search for an orthodox way to change > > the tree structure of a synthetic filesystem. > > Moving large real files is not my actual issue here. I'm wondering for a > > synthetic filesystem in which, when you move a folder in a special > > directory, something magic happens. > > As far as I can see, it is not possible with a 9p2000 fileservice, is it? > > i don't see why you can't make a magic directory that works that way. i agree with you that the magic is upto the fileserver to provide but more generally, i think it would be an interesting experiment for Giacomo Tesio to try to extend 9p. attempting to extend something is a great way to learn. Tmove of files on the same real FS can save a lot of unnecessary traffic. Tmove of dirs can save some traffic too. of course either type of move must be allowed to fail and must fail without any damage if an atomic move is not possible. i think unix and plan9 design reflects choices available in 70s & 80s and it certainly makes sense for someone not burdened with that history to reexamine the design choices. machines are much faster now so micro efficiency is less critical but latency is more so. and storage capacities have grown much faster than network bandwidth so opportunities for flinging less data about should be looked at. the unification of synthetic and real filesystems seems rather constraining now. storage is used much more creatively now and may be distributed. can there a generalized protocol for distributed storage? lots of ideas to play with!
Re: [9fans] wstat and atomic directory change
Il 05/Feb/2015 17:26 "erik quanstrom" ha scritto: > > I'm wondering for a > > synthetic filesystem in which, when you move a folder in a special > > directory, something magic happens. > > As far as I can see, it is not possible with a 9p2000 fileservice, is it? > > i don't see why you can't make a magic directory that works that way. > How? Which 9p message would trigger the magic? I just see 2 options: 1. Have a control file 2. Use custom wstat with full path to move But if both are wrong, I'm lost. Giacomo
Re: [9fans] wstat and atomic directory change
> All this reflections arise from the search for an orthodox way to change > the tree structure of a synthetic filesystem. > Moving large real files is not my actual issue here. I'm wondering for a > synthetic filesystem in which, when you move a folder in a special > directory, something magic happens. > As far as I can see, it is not possible with a 9p2000 fileservice, is it? i don't see why you can't make a magic directory that works that way. - erik
Re: [9fans] wstat and atomic directory change
> Also, in the absence of symbolic links, you don't get to choose > whether to migrate the "node" or the contents. Things get hairy, not > just complicated. sorry, we're going to have to disagree on this one. symbolic links are a huge pain in the butt. and to address your issue, both find and du have options to avoid traversing mount points. - erik
Re: [9fans] wstat and atomic directory change
> I'm not an advocate of Tmove in any way, but I can't really grasp the cons. > I'm sure that its omission was an explicit design choise, but where I > can read about the arguments that lead to such decision? It's a long time ago, but I seem to recall that the clinching argument hinged around mounted directories. Without making an a priori decision about each one, an atomic move is not viable. Remember that to the OS the difference between a local and a remote object is intentionally invisible. And then you need to add per-user and per-process namespaces... Lucio.
Re: [9fans] wstat and atomic directory change
> I mean we could simply introduce a new command "rename oldpath > newpath" that only works when both path share the same mount point. > This way the mv commands would keep the old "safe" semantic, while the > new command would protect the user to accidentally disclosure his data > to the world via the cloud. > But you don't know that somewhere further down the hierarchy the target isn't bound to some mount point. What happens then? You asked for atomic, but you can't have it without first descending the entire hierarchy and checking for difficult to detect conditions. >> Imagine a Tmove that transfers your >> entire disk contents to the cloud: would you like it to be perceived >> as trivial? What happens if you interrupt it? Worse, what happens if >> you can't interrupt it? > > I won't be drammatic: you can always unplug the enthernet! :-D > > Btw, I see the point. Well, did you spot that you may have the Internet mounted somewhere in your source directory? Occasionally I forget that I have my NetBSD server bound to my Plan 9 home directory and initiate a search on $home for some lost item. It takes a very long time to complete the search. Now, I may decide to move $home elsewhere... Also, in the absence of symbolic links, you don't get to choose whether to migrate the "node" or the contents. Things get hairy, not just complicated. Lucio.
Re: [9fans] wstat and atomic directory change
Actually I've found a 9 years old mail about Tmove: http://comp.os.plan9.narkive.com/xYi8Vg5d/9fans-fuse-bashing#post40 I'm not an advocate of Tmove in any way, but I can't really grasp the cons. I'm sure that its omission was an explicit design choise, but where I can read about the arguments that lead to such decision? Giacomo 2015-02-05 9:21 GMT+01:00 Giacomo Tesio : > 2015-02-05 5:13 GMT+01:00 : >> >> > But why we don't have Tmove for example? >> >> Because its semantics are much, much more complex and the users need >> to be aware of the difference. > > This shouldn't be so hard to obtain. > > I mean we could simply introduce a new command "rename oldpath > newpath" that only works when both path share the same mount point. > This way the mv commands would keep the old "safe" semantic, while the > new command would protect the user to accidentally disclosure his data > to the world via the cloud. > >> Imagine a Tmove that transfers your >> entire disk contents to the cloud: would you like it to be perceived >> as trivial? What happens if you interrupt it? Worse, what happens if >> you can't interrupt it? > > I won't be drammatic: you can always unplug the enthernet! :-D > > Btw, I see the point. > > > Giacomo
Re: [9fans] wstat and atomic directory change
2015-02-05 5:13 GMT+01:00 : > > > But why we don't have Tmove for example? > > Because its semantics are much, much more complex and the users need > to be aware of the difference. This shouldn't be so hard to obtain. I mean we could simply introduce a new command "rename oldpath newpath" that only works when both path share the same mount point. This way the mv commands would keep the old "safe" semantic, while the new command would protect the user to accidentally disclosure his data to the world via the cloud. > Imagine a Tmove that transfers your > entire disk contents to the cloud: would you like it to be perceived > as trivial? What happens if you interrupt it? Worse, what happens if > you can't interrupt it? I won't be drammatic: you can always unplug the enthernet! :-D Btw, I see the point. Giacomo
Re: [9fans] wstat and atomic directory change
2015-02-05 5:26 GMT+01:00 : > > But why we don't have Tmove for example? > > http://9front.org/img/tmove.jpg > > ROFTL! :-D
Re: [9fans] wstat and atomic directory change
> But why we don't have Tmove for example? http://9front.org/img/tmove.jpg sl
Re: [9fans] wstat and atomic directory change
> But why we don't have Tmove for example? http://9front.org/img/9tmove01.png sl
Re: [9fans] wstat and atomic directory change
> I'm wondering for a > synthetic filesystem in which, when you move a folder in a special > directory, something magic happens. "Magic" is overrated. Lucio.
Re: [9fans] wstat and atomic directory change
> But why we don't have Tmove for example? Because its semantics are much, much more complex and the users need to be aware of the difference. Imagine a Tmove that transfers your entire disk contents to the cloud: would you like it to be perceived as trivial? What happens if you interrupt it? Worse, what happens if you can't interrupt it? Lucio.
Re: [9fans] wstat and atomic directory change
Useless? No I did not mean this. (and sorry, I wasn't in any way sarcastic, I'm just trying to understand 9p's design a bit deeper) I mean that to realize a truly uniform interface, directories should be writable with Twrite, so that we would not need messages like Tcreate and Tremove (and possibly Twstat) and Tstat would just be an efficient shortcut (like it actually already is, since we can obtain the same info reading the parent directory). We could add a file by appending a stat entry to the parent directory and we could remove it by removing such entry. We could modify permissions, names and so on overwriting the entry. The problem would be: how to do such basic operations atomically and concurrently? That's why (I suppose) we have Tcreate, Tremove and Twstat. For praticality. But why we don't have Tmove for example? Probably the answer is in the distributed nature of Plan 9: as you said, namespaces would be at odds with such a message, complicating the library functions. However having to read and write a 10 GB file one msize after the other just to change its directory looks a bit costly, if both the origin and the destinations are in the same phisical disk. Moreover (if I've understood properly the protocol) to move a 10 GB file between two directories on a disk, you should have 10 GB free on that disk! All this reflections arise from the search for an orthodox way to change the tree structure of a synthetic filesystem. Moving large real files is not my actual issue here. I'm wondering for a synthetic filesystem in which, when you move a folder in a special directory, something magic happens. As far as I can see, it is not possible with a 9p2000 fileservice, is it? Giacomo 2015-02-04 20:23 GMT+01:00 erik quanstrom : > > you can consider doing it out of band; e.g. a new fossil console command. > > On Wed, Feb 4, 2015 at 6:30 AM Giacomo Tesio wrote: > > > > > And unfortunately I cannot figure out any alternative, either. :-( > > > > > > An interesting point is: why directory entries can be read but cannot > be > > > written? I mean we use Tcreate to create a new file in a dir, while to > be > > > uniform with files, we should use a Twrite, appending a corresponding > stat > > > structure at the end of a directory entry. > > > > > > However this solution probably has not been adopted to enable safe > > > concurrency in the file creation and removal. > > > > > > However this asymmetry shows that directories cannot be (pratically) > > > handled in the same way of files. > > right. directories are not the same as files. you're right that reading a > directory is useless. > > - erik > >
Re: [9fans] wstat and atomic directory change
> you can consider doing it out of band; e.g. a new fossil console command. > On Wed, Feb 4, 2015 at 6:30 AM Giacomo Tesio wrote: > > > And unfortunately I cannot figure out any alternative, either. :-( > > > > An interesting point is: why directory entries can be read but cannot be > > written? I mean we use Tcreate to create a new file in a dir, while to be > > uniform with files, we should use a Twrite, appending a corresponding stat > > structure at the end of a directory entry. > > > > However this solution probably has not been adopted to enable safe > > concurrency in the file creation and removal. > > > > However this asymmetry shows that directories cannot be (pratically) > > handled in the same way of files. right. directories are not the same as files. you're right that reading a directory is useless. - erik
Re: [9fans] wstat and atomic directory change
you can consider doing it out of band; e.g. a new fossil console command. On Wed, Feb 4, 2015 at 6:30 AM Giacomo Tesio wrote: > And unfortunately I cannot figure out any alternative, either. :-( > > An interesting point is: why directory entries can be read but cannot be > written? I mean we use Tcreate to create a new file in a dir, while to be > uniform with files, we should use a Twrite, appending a corresponding stat > structure at the end of a directory entry. > > However this solution probably has not been adopted to enable safe > concurrency in the file creation and removal. > > However this asymmetry shows that directories cannot be (pratically) > handled in the same way of files. > > > Giacomo > > > 2015-02-04 15:06 GMT+01:00 erik quanstrom : > >> > You are right. >> > But since protocol customizations goes against that same ideal, what's >> the >> > plan9 way of dealing with a feature X (in this specific case atomic file >> > moves)? >> > >> > >> > > this requires delving into the namespace to tell that the directories >> are >> > > still under the >> > > same mount, and some non-uniform operations. >> > > >> > >> > Good point. But thus, what are the alternatives? >> >> i wasn't holding out. i don't have any suggestions. >> >> - erik >> >> >
Re: [9fans] wstat and atomic directory change
And unfortunately I cannot figure out any alternative, either. :-( An interesting point is: why directory entries can be read but cannot be written? I mean we use Tcreate to create a new file in a dir, while to be uniform with files, we should use a Twrite, appending a corresponding stat structure at the end of a directory entry. However this solution probably has not been adopted to enable safe concurrency in the file creation and removal. However this asymmetry shows that directories cannot be (pratically) handled in the same way of files. Giacomo 2015-02-04 15:06 GMT+01:00 erik quanstrom : > > You are right. > > But since protocol customizations goes against that same ideal, what's > the > > plan9 way of dealing with a feature X (in this specific case atomic file > > moves)? > > > > > > > this requires delving into the namespace to tell that the directories > are > > > still under the > > > same mount, and some non-uniform operations. > > > > > > > Good point. But thus, what are the alternatives? > > i wasn't holding out. i don't have any suggestions. > > - erik > >
Re: [9fans] wstat and atomic directory change
> You are right. > But since protocol customizations goes against that same ideal, what's the > plan9 way of dealing with a feature X (in this specific case atomic file > moves)? > > > > this requires delving into the namespace to tell that the directories are > > still under the > > same mount, and some non-uniform operations. > > > > Good point. But thus, what are the alternatives? i wasn't holding out. i don't have any suggestions. - erik
Re: [9fans] wstat and atomic directory change
2015-02-04 4:51 GMT+01:00 erik quanstrom : > On Tue Feb 3 01:01:02 PST 2015, st...@quintile.net wrote: > > > the control file idea is a neat way of doing atomic moves. > > > > control files are a good way of dealing with all kinds of exceptional > behavior. > > but ... > > to be fair, this goes against the unix ideal of having uniform file > operations. > You are right. But since protocol customizations goes against that same ideal, what's the plan9 way of dealing with a feature X (in this specific case atomic file moves)? > this requires delving into the namespace to tell that the directories are > still under the > same mount, and some non-uniform operations. > Good point. But thus, what are the alternatives? Giacomo
Re: [9fans] wstat and atomic directory change
On Tue Feb 3 01:01:02 PST 2015, st...@quintile.net wrote: > the control file idea is a neat way of doing atomic moves. > control files are a good way of dealing with all kinds of exceptional behavior. but ... to be fair, this goes against the unix ideal of having uniform file operations. this requires delving into the namespace to tell that the directories are still under the same mount, and some non-uniform operations. so, it's not without a cost. - erik
Re: [9fans] wstat and atomic directory change
the control file idea is a neat way of doing atomic moves. this has been discussed before, my summary is its not something you need often to justify the pain of trying to implement it correctly - the directory locking has to be done with care to ensure it is all deadlock free. I do, very rarely, move directories around, generally I just take more care when creating them - knowing it will be a hassle to change later. I also, run fossil and venti, so the copies take no more disk space, only the directory entries. just my 2 cents worth. -Steve > On 3 Feb 2015, at 08:53, Giacomo Tesio wrote: > > Ok, got it. > > This annoing thread (sorry) was due to the fact that the only messages that > actually contains the "/" marker are Tauth and Tattach (in the aname). I > still think that using wstat with such marker to atomically move files among > accessible folders would not violate the protocol specification, but actually > it would break existing servers and that's is probably enough to define it as > an extension to the protocol (say 9P2000.a) so that clients can know if the > server supports this semantic or not. > However, as I previously said, I don't think that the world need a new 9p > variant :-D > > Now, what's the proper 9p way that a client could use to tell a syntetic > server to atomically move a syntetic file between syntetic folders? > I bet that the answer is a control file receiving rc like commands (or any > other custom, human readable protocol). Would it be correct? > > Something like this: > > Given > mountpoint/ > + first/ > + moveme.txt > + second/ > + atomically > > Doing > echo mv /first/moveme.txt /second/ > mountpoint/atomically > > We optain either the following or a Rerror: > mountpoint/ > + first/ > + second/ > + moveme.txt > + atomically > > > Is it the proper way to achieve such kind of operations? > > > Thanks for your patience... :-) > > > Giacomo > > 2015-01-30 23:49 GMT+01:00 Anthony Sorace : >> >> > On Jan 30, 2015, at 10:59 , Giacomo Tesio wrote: >> > >> > It surely would not be conformant to Plan 9 systems, but to the protocol? >> >> No. Joel has it right. Writing a server which allows / in names would mean >> that the "/" you're slipping into a name wouldn't always be a directory >> indicator or name separator. Think of it as the protocol accommodating >> systems which use some other marker. >> >> The relevant point is that the "name" in question (which, as you noticed, >> the protocol allows to contain / even though plan9 doesn't) is the name >> *within a directory*, not a full path name. walk(5) probably gives the best >> explanation of this, or perhaps the discussion of create in open(5). >
Re: [9fans] wstat and atomic directory change
Ok, got it. This annoing thread (sorry) was due to the fact that the only messages that actually contains the "/" marker are Tauth and Tattach (in the aname). I still think that using wstat with such marker to atomically move files among accessible folders would not violate the protocol specification, but actually it would break existing servers and that's is probably enough to define it as an extension to the protocol (say 9P2000.a) so that clients can know if the server supports this semantic or not. However, as I previously said, I don't think that the world need a new 9p variant :-D Now, what's the proper 9p way that a client could use to tell a syntetic server to atomically move a syntetic file between syntetic folders? I bet that the answer is a control file receiving rc like commands (or any other custom, human readable protocol). Would it be correct? Something like this: Given mountpoint/ + first/ + moveme.txt + second/ + atomically Doing echo mv /first/moveme.txt /second/ > mountpoint/atomically We optain either the following or a Rerror: mountpoint/ + first/ + second/ + moveme.txt + atomically Is it the proper way to achieve such kind of operations? Thanks for your patience... :-) Giacomo 2015-01-30 23:49 GMT+01:00 Anthony Sorace : > > > On Jan 30, 2015, at 10:59 , Giacomo Tesio wrote: > > > > It surely would not be conformant to Plan 9 systems, but to the protocol? > > No. Joel has it right. Writing a server which allows / in names would mean > that the "/" you're slipping into a name wouldn't always be a directory > indicator or name separator. Think of it as the protocol accommodating > systems which use some other marker. > > The relevant point is that the "name" in question (which, as you noticed, > the protocol allows to contain / even though plan9 doesn't) is the name > *within a directory*, not a full path name. walk(5) probably gives the best > explanation of this, or perhaps the discussion of create in open(5). > > >
Re: [9fans] wstat and atomic directory change
> On Jan 30, 2015, at 10:59 , Giacomo Tesio wrote: > > It surely would not be conformant to Plan 9 systems, but to the protocol? No. Joel has it right. Writing a server which allows / in names would mean that the "/" you're slipping into a name wouldn't always be a directory indicator or name separator. Think of it as the protocol accommodating systems which use some other marker. The relevant point is that the "name" in question (which, as you noticed, the protocol allows to contain / even though plan9 doesn't) is the name *within a directory*, not a full path name. walk(5) probably gives the best explanation of this, or perhaps the discussion of create in open(5).
Re: [9fans] wstat and atomic directory change
On Fri, Jan 30, 2015 at 10:59 AM, Giacomo Tesio wrote: > As far as I can read intro(5), it explicitly excludes slash as a valid > character for the Plan 9 OS, but it also explicitly states that "the > protocol has no such restriction". My reading is that a 9p2000 server might allow a filename of the form "b/c"; e.g. this tree is possible: a |-- b | +-- c | +-- d +-- b/c +-- d Note that "a" / "b" / "c" / "d" is distinct from "a" / "b/c" / "d". Plan 9, because of its Unix heritage, will not be able to deal with this, but the protocol has no such limitation. On the other hand, if you bypass the OS and issue a Walk to "a/b/c/d", there is a well-defined response required in the protocol -- either open the file with that (impossible) name, or fail. > Still, using this protocol "feature" to enable atomic directory change could > be useful in my use cases, but I don't want to build yet another 9p2000 > extension. It'll have to be an extension -- a technically incompatible one, though the incompatibilities will not ever happen. --Joel
Re: [9fans] wstat and atomic directory change
It surely would not be conformant to Plan 9 systems, but to the protocol? As far as I can read intro(5), it explicitly excludes slash as a valid character for the Plan 9 OS, but it also explicitly states that "the protocol has no such restriction". Be patient: I'm asking because this could be a typo in the intro(5) man page, or in my understanding of its phrasing. Still, using this protocol "feature" to enable atomic directory change could be useful in my use cases, but I don't want to build yet another 9p2000 extension. Giacomo 2015-01-30 15:13 GMT+01:00 erik quanstrom : > > Now, since the protocol does not restrict names (even if Plan 9 does it), > > I'm wondering if setting the name to a full path starting from root could > > be used to change atomically the directory of a file (given the write > > permission on both original and target directory). > > > > Obviously I'm not referring to Plan 9 file servers (I guess this would > be a > > non retrocompatbile change), but I'm considering if such interpretation > > would be wrong (according to the official specifications). > > > > A server supporting such behaviour could be considered a 9p2000 > conformant > > server? > > it would not be conformant. the intro explicitly excludes slash as a > valid character. > (unlike a dns zone.) and its more detailed explinations are meant to hold > for the > entire section. > > but you could still do it, as long as the file servers were the same. > > - erik > >
Re: [9fans] wstat and atomic directory change
> Now, since the protocol does not restrict names (even if Plan 9 does it), > I'm wondering if setting the name to a full path starting from root could > be used to change atomically the directory of a file (given the write > permission on both original and target directory). > > Obviously I'm not referring to Plan 9 file servers (I guess this would be a > non retrocompatbile change), but I'm considering if such interpretation > would be wrong (according to the official specifications). > > A server supporting such behaviour could be considered a 9p2000 conformant > server? it would not be conformant. the intro explicitly excludes slash as a valid character. (unlike a dns zone.) and its more detailed explinations are meant to hold for the entire section. but you could still do it, as long as the file servers were the same. - erik
