Hi Tim,
I'm back at Google Pittsburgh, and have been talking with them about ways that Google can help LSST. Now that Google is an institutional member, the possibilities seem to be opening up. One technology that came up is "bigtable", which sounds interesting to me in the context of our issues with updating the object table during the night. Take a look at http://labs.google.com/papers/bigtable-osdi06.pdf and see what you think.
Very interesting reading! Assuming that LSST manages to work out the details regarding using google's proprietary software (and getting some support), I think we should carefully examine where else the bigtable (and gfs) could be applicable in our system. The bigtable depends on gfs and chubby, so in practice we would have to deploy and manage all 3, and this is certainly going to be non-trivial. This seems like an overkill if we want to use it just for the real-time-alerts related Object table updates. If we find more places for bigtable/gfs, it would certainly make sense to try reusing what they developed (and tested!). If not, it would likely be more efficient to deploy a solution compatible with whatever we will be using to manage the bulk lsst data. (after all, there is no magic, they simply partition data, distribute load, cluster frequently accessed data together, compress data, keep very frequently accessed data on disk, avoid lock collisions and throw lots of hardware at the problem =) - they just put it all together into a solid system.
One might wonder whether this could be our solution to _all_ LSST peta-scale problems. I think this greatly depends on the future improvements in bigtable (like adding secondary indexes which they mentioned). As they admitted at the end, "they have no complex queries to execute", and I believe we will (although I have not really seen them so far =)
They would like from us a test problem that would give them a concrete example of what we need to do with the object table updates, and I told them we would come up with something at our meeting next week.
I've started to work on a document summarizing the load on the Object table. I expect to have something in the next few days.
thanks, Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
