Re: [9fans] Regarding 9p based protocols message framing
Perhaps initially: over an IP network, 9P used to run over IL. still does, including on the system i'm sending this from. What advantages does it have over TCP? Does it worths the effort of tailoring it back in, esp. for a fossil/venti site? -- - Yaroslav
Re: [9fans] Regarding 9p based protocols message framing
On Wed Mar 21 16:11:41 EDT 2012, yari...@gmail.com wrote: Perhaps initially: over an IP network, 9P used to run over IL. still does, including on the system i'm sending this from. What advantages does it have over TCP? Does it worths the effort of tailoring it back in, esp. for a fossil/venti site? it takes maybe 10 minutes to put it back in. 9atom has il still, since i still use it. i never took it out. i see why one would want to eliminate il and go all tcp, but il is simplier and has no problems that bother me operationally. also there are some fine points of the argument where i have a different opinion than some. of course i may also be motivated by not wanting to put tcp in the file server. ymmv. (in fact, i'd go further than il. why do we need the ip in il?) tcp otoh has a handful of bugs that bite me now and again. the biggest current one is that there appears to be a bug in the implementation that triggers a condition that looks as if nagle is disabled for some connections and early bytes in the window are missing. (infinite resequence queue, full of tinygrams.) this crashes boxes when the run out of kernel memory. i've worked around this by limiting the resequence queue to a reasonable (scaled with the window) number of packets as well as bytes. clearly this is just a hack. but until the trigger for this condition is understood, it'll have to do. - erik
Re: [9fans] Regarding 9p based protocols message framing
What advantages does it have over TCP? Significantly simpler implementation. Lower overhead, giving better speed over local links. Also, although not an advantage of IL per se, there are a few advantages to Plan 9 having another protocol in the stack, especially one which can run 9p. It helps guard against any accidental network-dependent creep. Does it worths the effort of tailoring it back in, It's not hard to get back in. Erik's 9atom has it. Whether it's worth it or not is hard to say. I like having it, but NAT boxes (and occasionally non-NAT routers, although that's more rare) can cause problems. It is slower over common wide- area links (lacking TCP's congestion control). esp. for a fossil/venti site? It's mandatory if you want to use kenfs; it's entirely optional in a fossil/venti (or cwfs or kfs or whatever) environment. It's sort of depends what you want from the system. Personally, I'd suggest doing it if you have multiple systems on the same local net for the pedagogical benefits alone. Anthony signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [9fans] Regarding 9p based protocols message framing
What advantages does it have over TCP? Significantly simpler implementation. Lower overhead, giving better speed over local links. it's also message based. so things like rx depend on a protocol like il that preserves 0-byte writes. - erik
Re: [9fans] Regarding 9p based protocols message framing
On Mar 21, 2012, at 16:27 , erik quanstrom wrote: (in fact, i'd go further than il. why do we need the ip in il?) I'd love to have an IP-free, 9p-capable network in the mix. Plan 9 had this, way back when, with nonet (?), but that didn't survive the removal of the streams system in the 3rd edition (I know you know this already,but I'm not sure how widely it's known). Anthony signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [9fans] Regarding 9p based protocols message framing
I'd love to have an IP-free, 9p-capable network in the mix. Plan 9 had this, way back when, with nonet (?), but that didn't survive the removal of the streams system in the 3rd edition (I know you know this already,but I'm not sure how widely it's known). i'm probablly missing something, but i don't see why your lan couldn't be ip-free. you may have to have a sacrifical super-nat machine that transplants, say, ssh/nonet onto ssh/ip. the more plan 9-y way would be to import the gateway. i'm not sure which would be more elegant. - erik
Re: [9fans] Regarding 9p based protocols message framing
thanks, that's great. the retransmission interval is probally off by 3/4 orders of magnitude for today's local networks. 50-500µs would be a decent static retransmit interval. (goes to show that timing sleeps in ms anachronistic.) - erik
Re: [9fans] Regarding 9p based protocols message framing
importing /net is possible. just recently replaced openwrt linux nat router with a plan9 machine importing its /net.alt to avoid problems with random tcp connection drops. wrote a simple socks server so windows can use it as a gateway too. even use bittorrent thru that thing with udp relay :) works nicely overall. some programs have problems with /net.alt tho. mainly ape programs wich require a bind /net.alt /net. wrote hg extension that hooks mercurials http object obfuscated interface and uses plan9 webfs (/mnt/web) instead, also makes hg use factotum indirectly thru webfs. only disadvantage is that one doesnt get proper errstr thru all that python/ape wrappery :( -- cinap
Re: [9fans] Regarding 9p based protocols message framing
just recently replaced openwrt linux nat router with a plan9 machine importing its /net.alt to avoid problems with random tcp connection drops. where's it importing /net.alt from? - erik
Re: [9fans] Regarding 9p based protocols message framing
gateway machine has two ethernet cards. /net.alt is the external internet facing side where i get ip over dhcp from cable modem. the /net is the internal local network. running import gateway /net.alt on random plan9 machine to import the internet. -- cinap
Re: [9fans] Regarding 9p based protocols message framing
Why was I puzzled: because as a non Plan9 user / developer, I usually think of the underlaying transport technology (be it sockets or 9p) as a stream of bytes without explicit framing. As I understand, 9P itself is designed to operate on top of a message-oriented transport; however, it has everything required to run over a stream, esp. message length at beginning of every message. Framing is done by the library: the read9pmsg routine performs as many reads as necessary to return a complete 9P message to the caller.
Re: [9fans] Regarding 9p based protocols message framing
On Tue, Mar 20, 2012 at 7:42 AM, Yaroslav yari...@gmail.com wrote: Why was I puzzled: because as a non Plan9 user / developer, I usually think of the underlaying transport technology (be it sockets or 9p) as a stream of bytes without explicit framing. As I understand, 9P itself is designed to operate on top of a message-oriented transport; however, it has everything required to run over a stream, esp. message length at beginning of every message. Framing is done by the library: the read9pmsg routine performs as many reads as necessary to return a complete 9P message to the caller. Perhaps initially: over an IP network, 9P used to run over IL. With 9P2000, IL was deprecated and 9P was most typically run over TCP. There was a very old message to 9fans (like, early 90's kind of old) that implied that IL was much more efficient than TCP on the wire, but it probably doesn't matter now. 9P itself is not a stream-oriented protocol, nor is it what one would generally call, 'transport technology.' - Dan C.
Re: [9fans] Regarding 9p based protocols message framing
Perhaps initially: over an IP network, 9P used to run over IL. still does, including on the system i'm sending this from. - erik
Re: [9fans] Regarding 9p based protocols message framing
On Tue, Mar 20, 2012 at 14:32, Dan Cross cro...@gmail.com wrote: 9P itself is not a stream-oriented protocol, nor is it what one would generally call, 'transport technology.' I would beg to differ on this subject... Because a lot of tools in the Plan9 environment expose their facilities as 9p file systems, but expose other semantics than that of generic files -- i.e. a contiguous stream of bytes from start to EOF -- like for example RPC semantic in case of factotum; thus I would say that 9p is used as a session layer in the OSI terminology. (But as in TCP/IP stack we don't have other layers between transport and application I would call it a transport layer in such a context.)
Re: [9fans] Regarding 9p based protocols message framing
On Tue, Mar 20, 2012 at 4:30 PM, Ciprian Dorin Craciun ciprian.crac...@gmail.com wrote: On Tue, Mar 20, 2012 at 14:32, Dan Cross cro...@gmail.com wrote: 9P itself is not a stream-oriented protocol, nor is it what one would generally call, 'transport technology.' I would beg to differ on this subject... Because a lot of tools in the Plan9 environment expose their facilities as 9p file systems, but expose other semantics than that of generic files -- i.e. a contiguous stream of bytes from start to EOF -- like for example RPC semantic in case of factotum; thus I would say that 9p is used as a session layer in the OSI terminology. (But as in TCP/IP stack we don't have other layers between transport and application I would call it a transport layer in such a context.) That's one way of looking at it. However, the file as a stream of bytes abstraction is mapped onto 9P at a higher layer; 9P itself is really about discrete messages. The canonical transport layer in TCP/IP is TCP. But we're arguing semantics at that point; regardless, I think you'd find you hold something of a minority view. - Dan C.
Re: [9fans] Regarding 9p based protocols message framing
I would beg to differ on this subject... Because a lot of tools in the Plan9 environment expose their facilities as 9p file systems, but expose other semantics than that of generic files -- i.e. a contiguous stream of bytes from start to EOF -- like for example RPC semantic in case of factotum; thus I would say that 9p is used as a session layer in the OSI terminology. (But as in TCP/IP stack we don't have other layers between transport and application I would call it a transport layer in such a context.) That's one way of looking at it. However, the file as a stream of bytes abstraction is mapped onto 9P at a higher layer; 9P itself is really about discrete messages. The canonical transport layer in TCP/IP is TCP. i think you can put a finer point on this. section 5 of the manual defines 9p by defining its messages, and the relationship between them. what it doesn't define is semantics. what does it mean to write n bytes at offset o. well, nothing in particular, other than you send the appropriate Twrite and get the matching Rwrite back. it's perfectly valid for the file server to ignore the write (/dev/null) or to halt and catch fire. file servers as a whole are unfortunately rather more ordinary. i think it's important to note (see /sys/doc/net.ps) that this would all be utter chaos if it weren't for conventions. plan 9 has strong conventions, but there aren't any cops. - erik
[9fans] Regarding 9p based protocols message framing
Hello all!
I'm reading through the nice `factotum` paper [1], and the section
describing the `rpc` file behaviour (section 2.6. Factotum
transactions) states:
Programs authenticate with factotum by writing a request to the rpc
file and reading back the reply; this sequence is called an RPC
transaction. Requests and replies have the same format: a textual verb
possibly followed by arguments, which may be textual or binary. [...]
Once started, an RPC conversation usually consists of a sequence of
read and write transactions.
[...]
PS - FS: start proto=apop role=server
FS - PS: ok
Now my puzzle was related to what does actually sequence of read
and write transactions mean. To be more clearly:
* the cited paragraph clearly involves two protocols: one is the
factotum RPC protocol which is layered on-top of the 9p protocol;
* if we take only the factotum RPC protocol (in abstraction of 9p)
we **might** have the impression that each request and response is
sent as an individual message whose framing is that of an text line,
and 9p is used merely as a transport; (but this framing is not clearly
expressed, see below...);
* thus if we take only this factotum RPC protocol into
consideration the implementor might have the impression that the
implementation should roughly be: keep receiving write requests until
we have an entire request (pseudo-C):
handle_write (context, data) {
append (context-buffer, data);
if (request_ready (context) {
handle_request (context);
}
return (succeed);
* but if we look at how it is implemented [2] it is actually: one
request and reply must map over exactly one 9p read or write
operation.
handle_write (context, data) {
if (parse_request (context, data))
return (handle_request (context));
else
return (failed);
Why was I puzzled: because as a non Plan9 user / developer, I
usually think of the underlaying transport technology (be it sockets
or 9p) as a stream of bytes without explicit framing. Meanwhile I
think that the Plan9 usage of 9p is:
* sometimes as a bidirectional message pipe (i.e. read == pull
message, write == push message, similar to what ZeroMQ library
offers);
* sometimes as an stateless RPC (like HTTP): open file, write
request data in maybe multiple write operations, read reply data in
maybe multiple read operations; (like the `ctl` file, as I guess `cat
ctl` makes one write per read line);
* sometimes as an un-seekable stream of data (like the `log` file);
Now I see here advantages and disadvantages... Advantages:
* the developer doesn't need to bother anymore with framing -- it
is done directly by the FS; (and gosh is this a relief, as every other
protocol invented its own framing: length followed by data, line feed
terminated, empty line terminated, random token ended, etc.)
* concurrent outstanding requests -- just call write in parallel;
(I've read the protocol and the specification doesn't say its
forbidden, but I've read something a few months ago on this mailing
list which suggested it should not be done???)
Disadvantages:
* the tools must be aware of this mapping between 1 request == 1
write operation; thus if for example my request would be composed of
multiple lines, and I use `cat ./some-file` as an interface; then
`cat` would read one line and immediately send it as an individual
write operation which of course is not respecting the protocol built
ontop of 9p;
Thus I would like to hear the position of the people on this
mailing list about:
* the explicitly mapping of one request to one 9p operation;
* the recommended usage patterns of 9p when building control
protocols ontop;
Thanks,
Ciprian.
[1] Security in Plan 9 -- http://plan9.bell-labs.com/sys/doc/auth.html
[2] http://plan9.bell-labs.com/sources/plan9/sys/src/cmd/auth/factotum/rpc.c
Re: [9fans] Regarding 9p based protocols message framing
* the tools must be aware of this mapping between 1 request == 1
write operation; thus if for example my request would be composed of
multiple lines, and I use `cat ./some-file` as an interface; then
`cat` would read one line and immediately send it as an individual
write operation which of course is not respecting the protocol built
ontop of 9p;
In general, control requests are not multiple lines (I presume by line
you mean a string of characters delimited by '\n'). As you say, each
request must be presented in a single 9p message, and this fits well
with the guarantee that 9p read/write messages are atomic [provided
tbe length is = iounit]. The first example in 2.6 of /sys/doc/auth is
misleading -- if you try to do literally what it says:
% cd /mnt/factotum
% cat ctl
key dom=bell-labs.com proto=p9sk1 user=gre
!password='don''t tell'
key proto=apop server=x.y.com user=gre
!password='bite me'
^D
you'll find you get a write error after the first line; each 'key'
command actually has to be sent as a single write. You can't even
get it to accept multiple lines in one write:
term% echo 'key dom=bell-labs.com proto=p9sk1 user=gre
!password=bite-me' /mnt/factotum/ctl
echo: write error: multiline write not allowed
In my experience, sending control requests as single atomic writes
is a natural way to work. With failed requests communicated as
write errors, this makes it simple to match the error status with
the request.
Multiline responses are somewhat more common, and there is some
inconsistency in whether responses also need to be read as a single
atomic operation. This can make it tricky, for example, to use the
read(1) command -- which reads one char at a time -- in a script
dealing with a device control interface.
For example, this loop works:
/dev/irqalloc while (x=`{read}) echo $x
but this doesn't:
/dev/ioalloc while (x=`{read}) echo $x
Re: [9fans] Regarding 9p based protocols message framing
On Fri, Mar 16, 2012 at 7:40 AM, Ciprian Dorin Craciun ciprian.crac...@gmail.com wrote: Why was I puzzled: because as a non Plan9 user / developer, I usually think of the underlaying transport technology (be it sockets or 9p) as a stream of bytes without explicit framing. Even outside Plan 9, this is not always the case. UDP, IP, ethernet, Unix datagram sockets, email, HTTP. They're all message-based protocols. Russ
Re: [9fans] Regarding 9p based protocols message framing
On Fri, Mar 16, 2012 at 19:36, Russ Cox r...@swtch.com wrote: On Fri, Mar 16, 2012 at 7:40 AM, Ciprian Dorin Craciun ciprian.crac...@gmail.com wrote: Why was I puzzled: because as a non Plan9 user / developer, I usually think of the underlaying transport technology (be it sockets or 9p) as a stream of bytes without explicit framing. Even outside Plan 9, this is not always the case. UDP, IP, ethernet, Unix datagram sockets, email, HTTP. They're all message-based protocols. Russ I am aware that there are other types of transports with other semantics. Nevertheless I guess that at least 90% of the existing protocols are built directly on top of stream based transports than datagram ones. (I'm not speaking of ubiquity / underlaying technology, or else we'd conclude that we use Ethernet in 99.999% of the cases.) And what is even more funny, is that I guess a good majority of the protocols are in the end message-based (as you've mentioned HTTP, SMTP, etc.) although they are built ontop of a stream based protocol. Furthermore 9p exposes a semantic more similar with TCP than with UDP, and only when you take into account the read / write atomicity you see it as a datagram protocol. My concert was that in general a programmer used to implement things ontop of stream based protocols was thought to expect that one read on the client is not in general one read on the server due to fragmentation, etc., and that he must frame his messages accordingly. Thus my question was if when working with 9p it is a good choice to do explicit framing (as in the case building ontop of TCP) or to take for granted the read / write atomicity. Ciprian.
Re: [9fans] Regarding 9p based protocols message framing
On Fri Mar 16 13:37:36 EDT 2012, r...@swtch.com wrote: On Fri, Mar 16, 2012 at 7:40 AM, Ciprian Dorin Craciun ciprian.crac...@gmail.com wrote: Why was I puzzled: because as a non Plan9 user / developer, I usually think of the underlaying transport technology (be it sockets or 9p) as a stream of bytes without explicit framing. Even outside Plan 9, this is not always the case. UDP, IP, ethernet, Unix datagram sockets, email, HTTP. They're all message-based protocols. of course inside plan 9, pipes are repect write boundaries, too. - erik
