Hi, So, we may not need all the static data, but I thought that it would be value added to be able to say that a new transient occured at the position of a known, but non-variable object (say a distant, barely resolved galaxy), or that a new transient appeared x arcsec offset from the nucleus of a rather well resolved galaxy. or that there was a transient at the position of a known M dwarf (perviously non-variable).
Kem > Hi, > > A few thoughts following our telecon: > > Jacek Becla wrote: > >> Keywords: DataAccWG >> >> Hi, >> >> I just had a quick word with Kem, here is the summary plus >> some estimates/comments I added. >> >> >> 1) What is the expected size of pre-cached archived data >> at the base camp? >> We are expecting to have roughly 100TB of image data, >> corresponding catalog data will probably be ~10% of that >> So it is ~10TB (upper bound). >> >> 2) How much of that data will be needed on average per image? >> ~5GB (upper bound) >> > I think we need a combination of deep object data, which does not have > time history (it is constructed from deep stacks of images over a long > time span), and shallower object data which does have time history. For > the deep object data, if we assume 50 galaxies per arcmin**2, we have > about 2e6 objects per image. We need to work on the object data > definition, but let's suppose that for each filter we keep three > different kinds of magnitudes and their associated errors. We will > have some kind of classification, and a variety of data quality flags. > There will also be detailed shape information, but we likely don't need > this at base, so I ignore that here. If each of those mags/errors is a > 32 bit float, the flags take 16 bits, and the classification 8 bits, and > we have 5 bands (U will not go deep), then we need 2e6*5 filters*(3 > mags*2 quantities/mag * 4 bytes/quantity + 3 bytes for flags) = 2.7 GB. > > The time dependent data will basically be limited to stars at roughly > 25th in V. The density on the sky is highly variable, but an average > value of about 2 per arcmin**2 is probably not too far off. This gives > us roughly 1e5 stars per image on average. I think we will need summary > information for each star, plus perhaps the most recent ten measurements > (in whichever bands we have them). Again, we suffer from not having > defined our object data well. But suppose we summarize the star with > 10 floats per band, and then have for each time point 2 floats plus 2 > bytes of quality flags. Then for each star we need 5*10*4 + 10*(2*4 + > 2) = 300 bytes. The total for stars is then 0.07 GB. > > So, given the assumptions about what we need, I certainly think that > Kem's number is in the right ballpark. However, I wonder if we really > need to prestage all this data for every image. After all, we are > operating on difference images at base, and even granted that the > detection are supplemented by some list of objects to always measure, we > will be needing object information for only a tiny fraction of the stars > in the image - perhaps a few percent. So, we need to fetch info for a > few * 1e3 objects from the total of 2e6 total (mostly deep) objects for > which we have data. Does this change our strategy? > > Tim > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
