Hi Tim at al,
With Andy's permission, I'm forwarding his comments regarding
the google's bigtable
Jacek
-------- Original Message --------
Subject: Re: interesting reading
Date: Wed, 11 Oct 2006 17:36:53 -0700
From: Andrew Hanushevsky <[EMAIL PROTECTED]>
To: Jacek Becla <[EMAIL PROTECTED]>
References: <[EMAIL PROTECTED]>
Hi Jacek,
I've now read the paper. I agree with your assessment that the solution
really was to add as much hardware as needed. The recovery protocol
still seems to have some issues in terms of scaling should a lot of
servers die at once (the case with a network partition). That may or
may not be a "real" problem at many sites. The key/data model is quite
simple and not at all clear it's workable for LSST queries. The other
interesting thing is that the model depends on GFS to provide the lower
level recoverability and shared access. In their model, it also does
not work incredibly well because they are forced to always read 64K
even when they need less than that amount. This, as they point out,
causes tablet servers to saturate should the required data not already
be cached. Seems like xrootd would do better for them here :-)
The only other issue is that the system is aware of all the tablets even
when the applications being run need access to an extremely small subset
of them (which is the case). So, I expect that startup and recovery
latency would be an issue in large systems. Anyway, it was interesting
to read that they discovered that the devil was in the details.
Should LSST use this? Hmmmm, as you point out, it's probably not
supported. Not clear if it's open source and it's been in operation
for just about a year (though, in all fairness, with high usage).
Can LSST live with simple key/value schema? Don't know.
Andy
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
