Re: [9fans] ideas for helpful system io functions

2010-01-05 Thread Steve Simon
> I'm thinking about whether it's worth to change uclibc in a way > that it allows to plug-in userland-vfs'es. Nothing new under the sun I'am afraid, however if you go ahead this might be interesting. http://www.cs.ncl.ac.uk/publications/articles/papers/399.pdf -Steve

Re: [9fans] ideas for helpful system io functions

2010-01-05 Thread Enrico Weigelt
* Tim Newsham wrote: > ps. if you wanted to hide this ugliness of passing a buffer and > fd to a child process instead of just passing an fd, you could > still solve it in userland without a syscall. Write a library > that does buffered IO. Include unget() if you like. Write the > library in a

Re: [9fans] ideas for helpful system io functions

2009-12-08 Thread matt
That's not what I meant by joining two fds. back seat coding, nice

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread roger peppe
2009/12/7 Nathaniel W Filardo : >> fd1 := open("/foo1", ORDWR); >> fd2 := open("/foo2", ORDWR); >> fd3 := fdjoin(fd1, fd2); >> >> what is going to happen? >> something has got to initiate the requests to actually >> shift the data, and it's not clear which direction the >> data will flow. > > "file

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Nathaniel W Filardo
On Mon, Dec 07, 2009 at 02:36:36PM +, roger peppe wrote: > 2009/12/7 Sam Watkins : > > I meant for example if a process is reading from its stdin a open file 'A' > > and > > writing to stdout the input of a pipe 'B', rather than looping and > > forwarding > > data it may simply "join" these t

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Francisco J Ballesteros
the idea is that if fs knows enough to handle partitions like we are accustomed to, then partitioning code can be removed from everywhere else (but for compat) and existing tools used to handle partitions (e.g., fdisk) very much like they are used now. Either way, It's not standalone, in one case

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread erik quanstrom
On Mon Dec 7 11:16:04 EST 2009, n...@lsub.org wrote: > It seems that changing a bit fs(3) can suffice and is generic > enough for all usages required. In the end it might result in code > removed instead of adding code, but time will tell. As of today, it's > only an experiment. not everyone who

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Francisco J Ballesteros
It seems that changing a bit fs(3) can suffice and is generic enough for all usages required. In the end it might result in code removed instead of adding code, but time will tell. As of today, it's only an experiment. On Mon, Dec 7, 2009 at 5:10 PM, erik quanstrom wrote: >> fs is already larger

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread erik quanstrom
> fs is already larger than it was, there's an experimental > ongoing version that knows enough of partitioning to help > usb and others on that respect. why not just use sdloop(3)? - erik

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread roger peppe
2009/12/7 Francisco J Ballesteros : > I think he wants copyfile + a kproc. yup, i was thinking of inferno's sys->stream(). but neither is in a position to do the kind of redundancy optimisation that sam was talking about, AFAICS. at least it can avoid copying by calling bread and bwrite.

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Francisco J Ballesteros
I think he wants copyfile + a kproc. On 07/12/2009, at 15:37, rogpe...@gmail.com wrote: 2009/12/7 Sam Watkins : I meant for example if a process is reading from its stdin a open file 'A' and writing to stdout the input of a pipe 'B', rather than looping and forwarding data it may simply "jo

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread roger peppe
2009/12/7 Sam Watkins : > I meant for example if a process is reading from its stdin a open file 'A' and > writing to stdout the input of a pipe 'B', rather than looping and forwarding > data it may simply "join" these two fds, and exit.  The OS will then do what > is > necessary to make sure the

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Sam Watkins
On Mon, Dec 07, 2009 at 12:24:05PM +, roger peppe wrote: > if you wanted it, an "fd join" driver could be simply > implemented in a similar way: > > bind '#j4.5' /mnt/joined > open /mnt/joined/data to get a (read-only) fd that satisfies reads from fd 4 > until eof, then fd 5. That's not what

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Charles Forsyth
> i wonder if there's a way of perverting fs(3) i made the comment fairly idly, so i shouldn't take it too seriously.

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Francisco J Ballesteros
Hmmm. That's what a cat device do, only that it does so by looking at the sizes and not at eof indications. Also, it depends on seek pos., which wont work for streams. Perhaps a streamcat, although I don't like to have cats and streamcats. Perhaps yet another option. fs is already larger than it

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread roger peppe
2009/12/7 Mechiel Lukkien : > i've attached devbuf.c and devjoin.c, as example (for inferno). [saw this just after i'd posted] that's funny - you even chose the same device character for devbuf! to be honest, your devbuf.c is almost synomous with a pipe. for buffer sizes of <64K, writes on a pipe

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Charles Forsyth
>bind '#j4.5' /mnt/joined > ... to get a (read-only) fd that satisfies reads from fd 4 >until eof, then fd 5. i wonder if there's a way of perverting fs(3)

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread roger peppe
2009/12/5 Bakul Shah : >       int newfd = fdfork(oldfd); i'm not sure that there needs to be a new syscall to enable this. a driver would be adequate. here's one possibility: the driver implements "buffered streams" - i.e. reads are lazy, but previous reads can be re-read. bind '#β4.8192' /mnt

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread erik quanstrom
> since file descriptors are so essential, it may help to have "tools" > to use them. yesterday evening i hacked up devbuf.c and devjoin.c > after reading this thread. both offer a file "new". for devbuf.c > you can write data to it, then later consume it (yes, you could just > use a pipe inste

Re: [9fans] ideas for helpful system io functions

2009-12-07 Thread Mechiel Lukkien
On Sat, Dec 05, 2009 at 08:24:45AM -1000, Tim Newsham wrote: > ps. if you wanted to hide this ugliness of passing a buffer and > fd to a child process instead of just passing an fd, you could > still solve it in userland without a syscall. Write a library > that does buffered IO. Include unget()

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Sam Watkins
On Sat, Dec 05, 2009 at 12:59:34PM -0800, Bakul Shah wrote: > You cut out the bit about buffering where I explained what I meant. Your idea seems good, so long as the OS buffers data and keeps it around until all readers have consumed it there would be no problem. This would be another possible s

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Bakul Shah
On Sat, 05 Dec 2009 15:27:02 EST erik quanstrom wrote: > > To be precise, both fds have their own pointer (or offset) > > and reading N bytes from some offset O must return the same > > bytes. > > wrong. /dev/random is my example. You cut out the bit about buffering where I explained what I me

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread erik quanstrom
> For disk based files and fifos there should be no > problem. there is no such distinction in plan 9. - erik

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread erik quanstrom
> To be precise, both fds have their own pointer (or offset) > and reading N bytes from some offset O must return the same > bytes. wrong. /dev/random is my example. - erik

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Bakul Shah
On Sat, 05 Dec 2009 15:03:44 EST erik quanstrom wrote: > > The OS support I am talking about: > > a) the fork behavior on an open file should be available > >*without* forking. dup() doesn't cut it (both fds share > >the same offset on the underlying file). I'd call the new > >syscal

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread erik quanstrom
> The OS support I am talking about: > a) the fork behavior on an open file should be available >*without* forking. dup() doesn't cut it (both fds share >the same offset on the underlying file). I'd call the new >syscall fdfork(). That is, if I do > >int newfd = fdfork(oldfd)

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Bakul Shah
On Sat, 05 Dec 2009 08:24:45 -1000 Tim Newsham wrote: > >> I can see two possible solutions for this, both of which would be useful i > n > >> my > >> opinion: > >> > >> - an "unread" function, like ungetc, which allows a program to put back > >> some > >>data that was already read to the

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Tim Newsham
I can see two possible solutions for this, both of which would be useful in my opinion: - an "unread" function, like ungetc, which allows a program to put back some data that was already read to the OS stdin buffer (not the stdio buffer). This might be problematic if there is a limit to

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Tim Newsham
I can see two possible solutions for this, both of which would be useful in my opinion: - an "unread" function, like ungetc, which allows a program to put back some data that was already read to the OS stdin buffer (not the stdio buffer). This might be problematic if there is a limit to th

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Skip Tavakkolian
>> I would like to pass the extra buffered data to the guy I am execing then let >> him read the rest directly from the socket, but I see no existing way to do >> that. > > httpd passes the headers and any left over buffer it has already read to > /magic > apps through a command line param. there

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Skip Tavakkolian
> I would like to pass the extra buffered data to the guy I am execing then let > him read the rest directly from the socket, but I see no existing way to do > that. httpd passes the headers and any left over buffer it has already read to /magic apps through a command line param. there's a functio

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread ron minnich
On Sat, Dec 5, 2009 at 9:01 AM, Francisco J Ballesteros wrote: > I mostly agree, but, if you read one char at a time it's likely you'll > become quite > slow, in general. Absolutely right. It's very application dependent. But for an httpd, I doubt that this slowness would matter. Anyway, I think

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Francisco J Ballesteros
I mostly agree, but, if you read one char at a time it's likely you'll become quite slow, in general. An external process providing `buffering' so you can seek back if you want, seems to me like a more general solution that does not require a kernel change. In any case, if I gave the impression th

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread ron minnich
On Sat, Dec 5, 2009 at 3:44 AM, Francisco J Ballesteros wrote: > If you insist on 'unreading', you could just put a front-end process that > keeps per-request data so that your external process can ask the > front-end for all the data again. The easiest way to implement unread is not to read in

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Sam Watkins
On Sat, Dec 05, 2009 at 08:26:20AM -0500, erik quanstrom wrote: > if you don't need to modify the data futher, then exec the guy who > does. This is my issue - when I want to exec, too much of the request data has already been read. I don't want to be calling read(fd, buf, 1) in a loop. I would l

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread erik quanstrom
On Sat Dec 5 00:12:53 EST 2009, lyn...@orthanc.ca wrote: > > Where FD passing is useful is to avoid that fork/exec overhead. > > Sorry -- brain in neutral. Where FD passing wins BIG is that the front-end > process doesn't have to do copy-through of all the data between the > network and the bac

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread erik quanstrom
On Sat Dec 5 03:11:09 EST 2009, s...@nipl.net wrote: > > the standard way of passing file descriptors is by fork/exec. > > this allows security is handled by the normal means. > > Erik/others, would you please give some feedback on my idea (a join call which > connects two fds together and disown

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Francisco J Ballesteros
I guess the question is, is this the easy way to address the problem you try to solve? Or is it a solution seeking for a problem? You could just forward the data to the new process. Is there a performance problem here? If you insist on 'unreading', you could just put a front-end process that keeps

Re: [9fans] ideas for helpful system io functions

2009-12-05 Thread Sam Watkins
> the standard way of passing file descriptors is by fork/exec. > this allows security is handled by the normal means. Erik/others, would you please give some feedback on my idea (a join call which connects two fds together and disowns them from the process). Passing fds around does not solve the

Re: [9fans] ideas for helpful system io functions

2009-12-04 Thread Lyndon Nerenberg
Where FD passing is useful is to avoid that fork/exec overhead. Sorry -- brain in neutral. Where FD passing wins BIG is that the front-end process doesn't have to do copy-through of all the data between the network and the back-end process.

Re: [9fans] ideas for helpful system io functions

2009-12-04 Thread Lyndon Nerenberg
the standard way of passing file descriptors is by fork/exec. this allows security is handled by the normal means. Where FD passing is useful is to avoid that fork/exec overhead. The apps I was working on had a relatively simple front-end process that would field requests that required data to

Re: [9fans] ideas for helpful system io functions

2009-12-04 Thread erik quanstrom
On Fri Dec 4 22:39:59 EST 2009, lyn...@orthanc.ca wrote: > > Another example, a little server that allows connections on a single port > > 443 > > for https and ssh. Ideally after reading the "GET" or ssh banner, it can > > just > > exec whichever server is needed (or fork and exec something li

Re: [9fans] ideas for helpful system io functions

2009-12-04 Thread Lyndon Nerenberg
My proposed type of CGI would have an advantage (?) that it presents a bidirectional socket to the script, rather than a file that was already read and saved to disk and a write-only socket. CGI chat over a single http connection for example would be possible (if the browser/client also supported

Re: [9fans] ideas for helpful system io functions

2009-12-04 Thread Sam Watkins
On Fri, Dec 04, 2009 at 08:36:29PM -0700, Lyndon Nerenberg wrote: > >Another example, a little server that allows connections on a single port > >443 for https and ssh. Ideally after reading the "GET" or ssh banner, it > >can just exec whichever server is needed (or fork and exec something like >

Re: [9fans] ideas for helpful system io functions

2009-12-04 Thread Lyndon Nerenberg
Another example, a little server that allows connections on a single port 443 for https and ssh. Ideally after reading the "GET" or ssh banner, it can just exec whichever server is needed (or fork and exec something like netcat). but in fact due to this "already read some data" problem, it has t

[9fans] ideas for helpful system io functions

2009-12-04 Thread Sam Watkins
I have two ideas for io functions that I think would be helpful, they are alternative options to solve a simple problem really. I don't know if plan 9 has any functions like these already. For example, when starting a CGI script for a POST request, a httpd reads the http headers but typically als