Keywords: DataAccWG
Attendees: Nicole, Russell, Ani, Maria, Ghaleb, Sergei, Ray, Kem, Jacek
staging data / detecting missing data
=====================================
Problem: need to cache certain subsets of data at the base
- need to have sense what to bring from the archive and
what is already pre-cached at the base
- need to be able to get data on the fly in case it is
needed but was not pre-cached (e.g. pipeline enters
unexpected path of processing, needs certain data we did
not expect it will need)
- how to determine if missing data is at the main archive
without frequently querying main archive?
Discussion:
- provenance could be lengthy
- the unit of ingest: ccd / image
- a flag per ccd: ingested ok/ingest failed
- "compress" it: if all ccds ok, then don't store all 201 "ok",
but just one per image
- so need need 2 flags (inside image metadata):
- image processed / not processed
- all ccds ingested ok / ingest failed for some ccds
- if failed then need to keep the list of failed
ccds in separate table
- but, is the ccd/image the right granularity - does it
help with precaching???
- asking each night for everything that we might need
is not an option: too expensive, don't have enough bandwidth
- can't track what is at the base / what is at the
main archive on object level - that it too much metadata
- need to find better granularity
- need fairly fine grain so that edge effect don't kill us
- ~few thousand sections, not just a few
- could leverage sky indexing mechanism (jhu / RA, decl, zoning)
- how much extra area do we want to allow ? (how far we want
to go beyond the border of the image?)
- 1 arc min extra?
- will come back to this topic at the next meeting...
schema
=======
UW waiting for Sergei to send schema file
- miscommunication, file sent some time ago, but maybe
not to the whole list
- Sergei will (re)-send right now
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
Reminder: we have the Database telecon
today at 11:00 PST.
Phone number: 866 330 1200
passcode: 300 2363
Agenda:
- schema (progress/update)
- staging data / detecting missing data
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
