2007/4/14, ron minnich <[EMAIL PROTECTED]>:
On 4/14/07, Lluís Batlle <[EMAIL PROTECTED]> wrote:
>
> Now I still cannot image what kind of files/directories 'ts' should
> serve to offer a good interface to the user, and how 'enqueuing'
> should work, getting a nice 'RPC' through that filesystem.
> I'll think of it, sure.
The server I wrote for Plan 9 was originally a port of the TNT work
described here:
http://arxiv.org/PS_cache/astro-ph/pdf/9912/9912134v1.pdf. The TNT
server maintained task state describing work to be done, and
mutiplexed that work to clients. One of the nice things about TNT, not
mentioned in the paper, is that it supports "third party" or middlemen
clients, i.e. clients can become servers and grab a bunch of tasks to
be handed out. This distribution can be important if you have enough
nodes, and turn tasks over very frequently. Also, of course, clients
can put new work back into the bag of tasks. The work to be done was
communicated to the server to/from clients via RPC (SunRPC). So, I
began to do a straight conversion of this Unix socket-based server to
Plan 9; about 10 minutes into the job, I realized I was not thinking
about the problem correctly.
I was almost certain that I should rethink the whole program again,
and although I may use some kind of RPC in unix (for avoiding linking
against p9p libs), I should be able to provide an API independant from
RPC or file servers.
But having some code written in that direction would make me more sure
of my words. :)
I thought about what I would do if starting from scratch, on Plan 9.
Plan 9 makes writing servers so easy that I went that route. I dumped
the RPC-based model and created a server that represented the state of
the work as directories and files. Basically, the server has three
directories, questions, working, and answers. You put questions into
the questions directory by opening and writing a file. To pass your
.....
I'd like to know what kind of file the "task submitter" should open.
Did you have several names in the questions directory, and each
submitter took one at random? It should be a name not colliding with
others in working or answers...
Or maybe the submitter opened a file with _any_ name (altough no 'ls'
would show it in questions), and that was created with the same name
in working?
One old problem with task bags related to multiple users. What if more
than one user writes the task bag? This has been solved in complex
ways on Unixen, with authentication happening on the server so that
only the 'right' people get to the socket. This is solved trivially in
Plan 9 -- each user, needing this service, starts a task bag server --
problem solved.
Also in my 'ts' each user has (in general) its own server. It depends
on the socket path they use. On agreement among users, they can use a
common socket with proper unix socket permissions in the filesystem.
The beauty of a server is that you can stop thinking in terms of RPC!
The "files" create a structure that is normally done in RPC. So, on
Plan 9, try to avoid falling into old Unix patterns. If you start
thinking in terms of a 'server' socket, and not a 9p server, and if
you start writing RPC code, the odds are good that you are not
approaching the problem correctly.
I agree. But RPC may be as awful as lazy I am, and the file structure
requires more thinking to find a nice solution - it will be probably
published to users :)
I ported a few HPC computations to my taskbag server; it was pretty
trivial. I *think* I left the code on sources (man 9fs to see what
this statement means) in the 9grid directory; you can look.
I looked at /n/sources/9grid, but I could only find some code for xcpu
there. Nothing named 'task bags' at first glance.
BTW, this sort of 'file-based' task bag idea has been done to death as
well, usually on NFS and with a huge boatload of scripts to go with
it. One system I saw, for MPEG encoding, had a 100-page (or so) script
to use NFS files for holding tasks. 99 pages were for dealing with the
problems that come with using NFS for this purpose ... not being able
to guarantee exclusive create, open, and remove makes these things
messy.
Well, my purpose on ts has never been multiuser queues for different
computers. I thought it for the case a user knows the
multitasking-limits of the resources of a single computer. Using hard
disk heavily in two applications may slowdown both. Using at once two
applications each requiring the amount of physical ram in the
computer, may also slow/bring down the system. That's what ts was
meant for.
It's not just the idea of using files to describe work that is so
effective; it's the fact that the operations on the files are under
control of your server, and hence they're not really files, but a
representation of the work, shown to you as files. This disctinction
is very important. It's why a Plan 9 task bag server can work, and the
NFS-based task bag servers fail, and fail badly.
Sure. I barely can believe someone tried such a thing over NFS.
It's nice to have you involved in Plan 9; welcome aboard. Don't
hesitate to ask questions. It will be nice to see what you create.
Having seen code in the operating system, read some works from Pike
and Kernighan, watched the discussions in this list from a long time,
it'd be a honor contributing a bit to the OS.
But I cannot guarantee code in short term. My university project is
entirely for another computing field and I should work on it, my job,
and other studies grab my time in own elected priority. I can't say
that I _need_ ts in plan9 (I mostly use p9p and not the OS). But it's
a direction I'd like to move on.
I'll also consider the nice and simple idea from John Stalker.
Thank you for your inspiring and couraging messages,
Lluís
