Hello all. I'm looking for some advice on whether Coda would be
appropriate for my situation. I have read the FAQ and docs, but it
seems that most people are using Coda for a different application and
all the docs have that in mind. It may be nobody uses Coda the way I'm
thinking because it's a stupid idea, so that's why I'm asking!
At present we have a small server farm which processes HTML logs for a
large number of websites (in the tens of thousands). The way this works
is we have seven processing servers which do nothing but parse logs and
generate reports. The reports, once created, are saved forever to a
storage server (via NFS) which is directly attached to a big (~1.9TB) array.
There are two problems here. First, if the storage server goes offline
for whatever reason (like if NFS decides to flake out for a few
seconds), all the processing servers hang and have to be power cycled.
This is Very Bad. And second, the processing servers all need to get
the actual logs and store them on a local filesystem (because NFS is way
too slow). These logs can get very very big so if a bunch of requests
arrive all at once - like when the monthly reports are automatically
generated - they tend to run out of disk space and die.
I am thinking that Coda might be able to provide a solution to these
problems - a better one than the one we are using now, anyway, which is
"throw more hideously expensive servers at the problem and hope it goes
away."
My rough-sketch thought process is that, naturally, the storage server
would still provide all the disk space. Each Coda client (the
processing servers) would make a cache about the size of the partition
it's using now for its temporary files (anywhere from 50-140GB).
The first problem would be solved, or at least mitigated, because the
coda clients could still do their thing if the storage server crapped
out for a while. Most of the time the processing servers run at 40%
disk capacity so that should leave enough space for at least an hour or
two of disconnected operation. If the array fails or something, we're
pretty screwed anyway, but at least we could process requests for the
time it takes to reinitialize the thing.
The second problem would be, again, solved or at least mitigated because
the coda clients would have "emergency backup" storage. The temporary
files would be written to /coda and go in the cache. Since those files
only have lifetimes of a few minutes and are never requested by other
clients in the farm, they should never need to go over the network
unless the cache fills up. If the cache does fill up, which it will at
least once a month, performance will degrade (significantly) but at
least the requests will still be processed. Once the backlog gets
handled (takes about a day) the caches will clear out and everything
will go back to normal. If the storage array fills up to the point
where we're running out of disk space again, we can add another (much
smaller) one just for this temporary storage. This way we could avoid
dropping huge sums on a quad-CPU box, which is what we're doing now,
just to add storage capacity.
Now after that lengthy story and rationale, my first question is
obvious: Is my reasoning correct? Is this something Coda can do, even
though most people aren't using it quite like I want to?
Second question should also be pretty obvious and it's a FAQ: Is Coda
reliable enough to be used for this? I know the FAQ says "no" but our
current solution is already terribly unreliable (we lose a ton of
reports every month due to the disk space problems alone; I have no hard
figures but I'd estimate up to an eighth of the monthly requests are
lost, forcing our customers to request their reports manually a day or
week later). As long as Coda can more-or-less guarantee that the
archived reports on the storage array won't get trashed... If the major
reliability concern is having to restart coda processes occasionally, we
can do that.
Third question is related to the first two: If Coda is not, for whatever
reason, appropriate for this, is there something similar which is? I've
looked at Lustre, and probably will continue looking at it, but it seems
geared towards much larger clusters than ours. It may also be much less
of a "drop-in" replacement than Coda, which is by now a standard part of
the free Unix-like OSes.
Thanks in advance for any tips or pointers, and to anyone who actually
read this whole thing, I admire your stamina!
