Hello all, Coincidentally, I just had a conversation about this with German Schumacher, who is on the Telescope Team and the lead on the control system and scheduler. I made him aware that we will need to see the scheduler nightly plan prior to obeserving and updates to it in near reali-time, so that we can pre-fetch data. This is an interface between the OCS/Scheduler and the Data Management control and management middleware that needs to be documented in our model. We didn't address the question of what data needs to be pre-fetched, that is a pure DM problem.
I also note that there are discussions about the bandwidth limitations relative to the pre-fetch and just want to remind everyone that as I understand it, the bandwidth we will have from base to archive is full duplex, that is, we get 5Gbps in each direction. Jeff > From: [EMAIL PROTECTED] > Reply-To: [EMAIL PROTECTED], LSST Data Management <[email protected]> > Date: Mon, 17 Apr 2006 17:55:51 -0400 (EDT) > To: [email protected] > Subject: Re: [LSST-data] Notes from Database telecon Wed Apr 12 > > Hello Jacek and DataAccWG. > > Perhaps the following statement is trivial or you have already > discussed it, but here is my take on this pre-caching dilemma: > > ---- For LSST, the 'scheduling' system for the observatory will > represent a different paradigm from current telescopes. We will > not only be scheduling the pointings of the telescope (plus > filter changes, etc.), but also the scheduling should include > the data management activities: pre-fetch and cached specific data > and data products and metadata and catalogs that are needed for > near-real-time data processing. The scheduling system is thus > a full end-to-end observatory scheduling system. With this view, > and hopefully with a decent implementation, the challenge areas > that you list become part of the solution and not part of the problem: > >> Problem: need to cache certain subsets of data at the base >> - need to have sense what to bring from the archive and >> what is already pre-cached at the base >> - need to be able to get data on the fly in case it is >> needed but was not pre-cached (e.g. pipeline enters >> unexpected path of processing, needs certain data we did >> not expect it will need) >> - how to determine if missing data is at the main archive >> without frequently querying main archive? >> >> - asking each night for everything that we might need >> is not an option: too expensive, don't have enough bandwidth >> >> - can't track what is at the base / what is at the >> main archive on object level - that it too much metadata >> >> - need to find better granularity >> >> - need fairly fine grain so that edge effect don't kill us >> - ~few thousand sections, not just a few >> >> - could leverage sky indexing mechanism (jhu / RA, decl, zoning) >> >> - how much extra area do we want to allow ? (how far we want >> to go beyond the border of the image?) >> - 1 arc min extra? > > > Finally, as part of "scheduling", you apply certain rules regarding > the time at which certain data products are pre-fetched and cached > (e.g., one day in advance, or N hours in advance, or whatever). > All scheduling systems have these types of dependencies and > subtasks built into the schedules. (My decade on Hubble trained > me to think about the scheduling system as a *SYSTEM*!) > > - Kirk > > >> Date: Wed, 12 Apr 2006 16:06:48 -0700 >> From: Jacek Becla <[EMAIL PROTECTED]> >> Subject: [LSST-data] Notes from Database telecon Wed Apr 12 >> To: LSST Data Management <[email protected]> >> >> Keywords: DataAccWG >> >> >> Attendees: Nicole, Russell, Ani, Maria, Ghaleb, Sergei, Ray, Kem, Jacek >> >> >> >> staging data / detecting missing data >> ===================================== >> >> Problem: need to cache certain subsets of data at the base >> - need to have sense what to bring from the archive and >> what is already pre-cached at the base >> - need to be able to get data on the fly in case it is >> needed but was not pre-cached (e.g. pipeline enters >> unexpected path of processing, needs certain data we did >> not expect it will need) >> - how to determine if missing data is at the main archive >> without frequently querying main archive? >> >> >> Discussion: >> >> >> - provenance could be lengthy >> >> >> - the unit of ingest: ccd / image >> - a flag per ccd: ingested ok/ingest failed >> - "compress" it: if all ccds ok, then don't store all 201 "ok", >> but just one per image >> - so need need 2 flags (inside image metadata): >> - image processed / not processed >> - all ccds ingested ok / ingest failed for some ccds >> - if failed then need to keep the list of failed >> ccds in separate table >> - but, is the ccd/image the right granularity - does it >> help with precaching??? >> >> >> >> - asking each night for everything that we might need >> is not an option: too expensive, don't have enough bandwidth >> >> - can't track what is at the base / what is at the >> main archive on object level - that it too much metadata >> >> - need to find better granularity >> >> - need fairly fine grain so that edge effect don't kill us >> - ~few thousand sections, not just a few >> >> - could leverage sky indexing mechanism (jhu / RA, decl, zoning) >> >> - how much extra area do we want to allow ? (how far we want >> to go beyond the border of the image?) >> - 1 arc min extra? >> >> >> - will come back to this topic at the next meeting... >> >> >> >> schema >> ======= >> >> UW waiting for Sergei to send schema file >> - miscommunication, file sent some time ago, but maybe >> not to the whole list >> - Sergei will (re)-send right now >> >> >> Jacek >> >> >> >> >> >> Jacek Becla wrote: >>> Keywords: DataAccWG >>> >>> Hi, >>> >>> Reminder: we have the Database telecon >>> today at 11:00 PST. >>> >>> Phone number: 866 330 1200 >>> passcode: 300 2363 >>> >>> Agenda: >>> - schema (progress/update) >>> - staging data / detecting missing data >>> >>> Jacek >> >> _______________________________________________ >> LSST-data mailing list >> [email protected] >> http://www.lsstmail.org/mailman/listinfo/lsst-data > > ----------------------------------------------------------------------------- > Dr. Kirk D. Borne > NASA Goddard Space Flight Center, SSDOO Program Manager, QSS Group Inc. > and George Mason University, Associate Research Professor, College of Science > <mailto:[EMAIL PROTECTED]> Tel. +1-301-286-0696 Fax: 301-286-1771 > Staff page: http://rings.gsfc.nasa.gov/~borne/ > US Virtual Observatory: http://www.us-vo.org/ > Large Synoptic Survey Telescope: http://www.lssto.org/ > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
