Thanks, this and Jacek's CPU for queries analysis are exactly what we need to kick-start the discussions at the TechAssessWG on Monday.
> From: Chris Smith <[EMAIL PROTECTED]> > Reply-To: LSST Data Management <[email protected]> > Date: Fri, 21 Jul 2006 09:49:33 -0400 > To: LSST Data Management <[email protected]> > Subject: Re: [LSST-data] Archive/Data Access Center computing and networking > > > Hi all, > > From my notes at the meeting, updated to be consistent with Don's > minutes (pertinent section below), I had > > BASELINE > - have 90TFlops at archive center > - maintenance: 10 TFlops > - real time processing: 25 TFlops (only 8-12 hrs) > - planned reprocessing: 25 TFlop (grows over time) > - query: 5-10 TFlops > - deep detection: 5 TFlops > leaves > - 25 TFlops on demand reprocessing > > But of course many of these (planned reprocessing in particular) are > functions of TIME. So I've tried to capture some of this in a > spreadsheet (attached). > > Here's the description: > > - First of all, the average observing night (including some twilight) > at Pachon is about 10hrs. Since the 25TFlops at the base facility is > sized to process data in real time, that's actually a need of only > 10.4 TFlops spread over 24 hours. > > - I've changed the "maintenance" to a *fraction* of the total CPU > instead of a fixed 10 TFlops. Is this correct? > > - The deep detection is assumed not to scale with time. While it is > true that just the *detection* will be on stacked (i.e., single) > images, the process of *stacking* images WILL scale over time, > because we have more images to stack. I haven't taken this into > account. > > - REPROCESSING: As we discussed, the reprocessing will scale with > time. I've postulated TWO components to this: > - an "initial reprocessing" load, which simply acknowledges that > as we start the survey we'll want to rerun the pipeline MANY times to > work out bugs and reprocess after bug fixes. > - a "routine reprocessing" which starts in year two and simply > scales linearly as the data volume grows to meet the requirement that > in any given year, we reprocess ALL previous data > > - QUERY LOAD: > - I've assumed some simple arbitrary functional load, with an up- > front peak as both the project and the users check out the system > - Jacek mentioned he was going to take a crack at this. I'd > encourage guessing at a *functional* form of the demand. Perhaps > this function could be informed by the query load as a function of > time seen by SDSS? > > - ON-DEMAND REPROCESSING > - with no reduced images online, we've got to reprocess to serve > users reduced images. While the catalog will be the "gold mine" for > science, there will a significant user community who want or need the > *images*. I've broken the load into three categories > - user verification: hopefully users, and/or LSST staff, will do > spot checks on the reductions EVEN IF they are using catalogs. This > will be something that peaks sometime early in the survey and then > tails off as we all learn to trust the LSST pipeline and resulting > catalog products > - reduced images: this is simply the load for reproducing > specific reduced images. I've made this a ramp up as the survey > starts, plateauing at something less than the "on demand" we agreed > upon in the telecon. > - transients: some fraction of the transients will require > detailed investigation of what was there beforehand, investigation at > a level that the information in catalogs won't be sufficient (obvious > example: we detect a hostless transient... there's nothing at that > spot before, even in the stacked images. But forced photometry at > that point could turn up a series of 2 sigma events which weren't > detected previously). I've postulated that this is also a function > of time, since some users will want to run the query "Give me the > forced photometry at exactly this spot for data going back N years". > If the user interested in the transient at this point waits until > next year, then this forced photometry may be done in the routine > reprocessing, but astronomers working on transients are often not > that patient. ;-) > > I've stopped at this point to open the floor for discussion. I think > it would be useful to come up with and agree upon some reasonable > functional forms for the query load and on-demand load, even if they > are way oversimplified, so we have some idea of how the CPU load > might track over time. > > As you can see, with the basic assumptions I've made here, the > 90TFlops at the Archive center is sufficient only out to around year 5. > > And you'll also note I've run "DM operations" out to beyond the 10yr > lifetime of the survey. If I were a reviewer, I'd nail LSST if they > didn't at least mention this, since the scientific results we're > claiming LSST will produce will not emerge until at least 1-2 years > after we have the *complete* data set! Perhaps we don't have to > "budget" for it, but I think acknowledging it will be a good idea. > > Cheers, > Chris > > > On Jul 19, 2006, at 5:44 PM, Don Dossa wrote: > >> From the tech assess concal -7-17-2006 >> >> The discussion centered around two main topics: Archive computing >> needs and network bandwidth between the archive site (AC) and data >> access center (DAC). >> >> We assume the astronomical image pipelines are not drastically >> changing but the query load imposed on the archive and data center >> sites can be variable and require additional computations, disk IO, >> and network bandwidth. >> >> Given as base computing requirement of 25 TF, then we assume an >> archive compute requirement of 90 TF of which about 80 TF should be >> available. This is allocated as 25 TF to reprocess the nightly raw >> images which are delivered over 24 hours. An allocation of 25 TF >> is for on-demand reprocessing but some of this load could be >> shifted to the researcher's home institution. Another 5 TF is >> needed for deep detection processing, and 5-10 TF to handle >> queries. The remaining 25+ TF is used to reprocess years of data. >> This personally seems low since that implies 3-4 nights of images >> are done per day. Other than the processing of each night¹s >> images, we expect the compute requirements to constantly increase >> over time. > > ======================================================================== > = > Dr. R. Chris Smith EMAIL: [EMAIL PROTECTED] > National Optical Astronomy Observatory > Cerro Tololo Inter-American Observatory > 950 N. Cherry Ave. Casilla 603 > Tucson, AZ 85719 La Serena, CHILE > Office: 520-318-8555 FAX: 520-318-8170 > WWW: http://www.ctio.noao.edu/~chris > ======================================================================== > = > > > > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
