Hello Jacek and DataAccWG. Perhaps the following statement is trivial or you have already discussed it, but here is my take on this pre-caching dilemma:
---- For LSST, the 'scheduling' system for the observatory will represent a different paradigm from current telescopes. We will not only be scheduling the pointings of the telescope (plus filter changes, etc.), but also the scheduling should include the data management activities: pre-fetch and cached specific data and data products and metadata and catalogs that are needed for near-real-time data processing. The scheduling system is thus a full end-to-end observatory scheduling system. With this view, and hopefully with a decent implementation, the challenge areas that you list become part of the solution and not part of the problem: > Problem: need to cache certain subsets of data at the base > - need to have sense what to bring from the archive and > what is already pre-cached at the base > - need to be able to get data on the fly in case it is > needed but was not pre-cached (e.g. pipeline enters > unexpected path of processing, needs certain data we did > not expect it will need) > - how to determine if missing data is at the main archive > without frequently querying main archive? > > - asking each night for everything that we might need > is not an option: too expensive, don't have enough bandwidth > > - can't track what is at the base / what is at the > main archive on object level - that it too much metadata > > - need to find better granularity > > - need fairly fine grain so that edge effect don't kill us > - ~few thousand sections, not just a few > > - could leverage sky indexing mechanism (jhu / RA, decl, zoning) > > - how much extra area do we want to allow ? (how far we want > to go beyond the border of the image?) > - 1 arc min extra? Finally, as part of "scheduling", you apply certain rules regarding the time at which certain data products are pre-fetched and cached (e.g., one day in advance, or N hours in advance, or whatever). All scheduling systems have these types of dependencies and subtasks built into the schedules. (My decade on Hubble trained me to think about the scheduling system as a *SYSTEM*!) - Kirk > Date: Wed, 12 Apr 2006 16:06:48 -0700 > From: Jacek Becla <[EMAIL PROTECTED]> > Subject: [LSST-data] Notes from Database telecon Wed Apr 12 > To: LSST Data Management <[email protected]> > > Keywords: DataAccWG > > > Attendees: Nicole, Russell, Ani, Maria, Ghaleb, Sergei, Ray, Kem, Jacek > > > > staging data / detecting missing data > ===================================== > > Problem: need to cache certain subsets of data at the base > - need to have sense what to bring from the archive and > what is already pre-cached at the base > - need to be able to get data on the fly in case it is > needed but was not pre-cached (e.g. pipeline enters > unexpected path of processing, needs certain data we did > not expect it will need) > - how to determine if missing data is at the main archive > without frequently querying main archive? > > > Discussion: > > > - provenance could be lengthy > > > - the unit of ingest: ccd / image > - a flag per ccd: ingested ok/ingest failed > - "compress" it: if all ccds ok, then don't store all 201 "ok", > but just one per image > - so need need 2 flags (inside image metadata): > - image processed / not processed > - all ccds ingested ok / ingest failed for some ccds > - if failed then need to keep the list of failed > ccds in separate table > - but, is the ccd/image the right granularity - does it > help with precaching??? > > > > - asking each night for everything that we might need > is not an option: too expensive, don't have enough bandwidth > > - can't track what is at the base / what is at the > main archive on object level - that it too much metadata > > - need to find better granularity > > - need fairly fine grain so that edge effect don't kill us > - ~few thousand sections, not just a few > > - could leverage sky indexing mechanism (jhu / RA, decl, zoning) > > - how much extra area do we want to allow ? (how far we want > to go beyond the border of the image?) > - 1 arc min extra? > > > - will come back to this topic at the next meeting... > > > > schema > ======= > > UW waiting for Sergei to send schema file > - miscommunication, file sent some time ago, but maybe > not to the whole list > - Sergei will (re)-send right now > > > Jacek > > > > > > Jacek Becla wrote: > > Keywords: DataAccWG > > > > Hi, > > > > Reminder: we have the Database telecon > > today at 11:00 PST. > > > > Phone number: 866 330 1200 > > passcode: 300 2363 > > > > Agenda: > > - schema (progress/update) > > - staging data / detecting missing data > > > > Jacek > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data ----------------------------------------------------------------------------- Dr. Kirk D. Borne NASA Goddard Space Flight Center, SSDOO Program Manager, QSS Group Inc. and George Mason University, Associate Research Professor, College of Science <mailto:[EMAIL PROTECTED]> Tel. +1-301-286-0696 Fax: 301-286-1771 Staff page: http://rings.gsfc.nasa.gov/~borne/ US Virtual Observatory: http://www.us-vo.org/ Large Synoptic Survey Telescope: http://www.lssto.org/ _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
