Hello all, I am concerned about double-counting on the backup storage we need in case of down-stream failure. Here is my take:
Per our TechAssessWG, we can keep two backup sets on the mountain while we are sending one to the base. We keep the first mountain copy until we confirm that the data has reached the Base. Then we can delete one of the Mountain copies. When the data has reached the Archive Center, we have 2 copies at base and archive, so we can delete the second mountain backup. This should be within 24 hours, but could be as long as 48 hours, per our MTTRs. Similarly, once we have confirmed that the data has reached the Data Access Center, we then have 2 copies, at the Archive Center and DAC, so we can delete the Base copy. In general, we only need one copy at any location (except the mountain) because there is another copy at the PREVIOUS UPSTREAM LOCATION being cached. Jeff > From: Ray Plante <[EMAIL PROTECTED]> > Reply-To: Ray Plante <[EMAIL PROTECTED]>, LSST Data Management > <[email protected]> > Date: Mon, 31 Jul 2006 15:14:28 -0500 (CDT) > To: LSST Data Management <[email protected]> > Subject: Re: [LSST-data] TechAssessWG > > On Fri, 28 Jul 2006, Don Dossa wrote: >> 1. Number CPUs for long term non-time critical science. Is this the >> design >> where we tell the users - "here is your data kid, no dont bother us" > > No. The 200 nodes for "extended processing" was intended for: > o non-nightly pipelines (e.g. co-add pipeline, deep-detection) > o reprocessing > > Of course, the recent analysis that Chris has been leading should inform > modifications to this allocation. > >> 2. DB ingest needs 128 CPUs. I remember seeing a plot by Sergei that >> showed the ingest rate ran on a single CPU. I also recall part of >> his plot of ingest rate leveling off if some DB function was included. >> I don't remember what that option was, but is it one we care about? > > I'm not sure that 128 CPUs is consistant with the current estimates for > ingest. (Jacek?) Currently, we expect ingest to only require a few > nodes, yes? > >> 3. File servers are shown as 64x8 CPUs. Why do we think we need more >> than 10% of the system to be file servers? I'm sure we are disk bound >> and not CPU bound. > > Let's put aside the number of CPUs for the moment. I believe we > origionally suggest fewer cores per node, but it wasn't worth the hassle > to calculate the cost that way. > > The more important thing (when we estimated it based on current > practices at NCSA) is the number of controllers you would need to > drive to handle that much disk space; that is, the number of nodes was the > more important number. > > Meanwhile, Jim Gray's recent analysis, if I'm recalling this right, > suggested we can drive 60 disks per core before becoming CPU-bound. This > suggests 8 servers would cut it. I'll note htat he had some different > projections for the sizing. > >> 4. As data flows in from the base camp, we need to consider how to >> reliably save the data. Of course we immediately write the data >> to disk at the archive site. Where do we put the second (backup) >> copy? It seems like we should immediately transmit the data from >> the systems at the network interface to the mirror site. > > That would be the plan. We would ensure that we have 2 copies at the > archive center until we have ensured replication to the data center. > >> For >> reliability purposes, I suggest the systems closest to the incoming >> network (lets call these buffer systems) each have 2 network >> interfaces and the outgoing network connections are different from >> the incoming network. > > Okay. > >> 5. How is a database backup done? I presume we cant take the >> database offline while we back it up. Do we run a duplicate >> database at the mirror site and update both databases simultaneously? > > We would regularly replicate new records from the archive center database > to the data center. > >> 6. Reliability/uptime requirements. I think Jeff and I think about >> these >> numbers in very different ways. If we want 99.9% uptime, we need >> to realize that 'uptime' means that either the entire system is down >> or some portion has failed that results in an unnacceptably degraded >> operation. What is a complete listing of all uncorrelated failures >> that can cause an unacceptable reliability and how can we minimize >> these? Let's list just a few obvious failure modes. >> >> 1. Network systems connecting archive to outside world. >> 2. A failure of any one of the 4 IB switches in the AC cluster. >> 3. Control system software (We will let Jeff estimate this number). >> >> How many non-redundant failure modes of this type do we have? What >> are the MTBF and MTTR numbers for each one? We need to think about >> what we require from our system components to achieve a 99.9% uptime. > > We thought some about this when we made the initial plan; however, I > probably need to go back and re-assess. > > cheers, > Ray > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
