Hi, Got it. Staging the queries that way probably would be faster and more efficient.
cheers, Kem > Hi Kem, > > I am in complete agreement! I guess I wasn't clear enough in what I > was proposing. I am not suggesting that we ignore the deep detection > data in alerting at base. What I *am* suggesting is that it may be > more efficient to query that data for the relatively small number of > detected transients rather than to blindly load it all into memory for > every frame. > > Tim > > Kem Cook wrote: > >>Hi, >> >>So, we may not need all the static data, but I thought that it would be >>value added to be able to say that a new transient occured at the >> position >>of a known, but non-variable object (say a distant, barely resolved >>galaxy), or that a new transient appeared x arcsec offset from the >>nucleus of a rather well resolved galaxy. or that there was a transient >> at >>the position of a known M dwarf (perviously non-variable). >> >>Kem >> >> >> >>>Hi, >>> >>>A few thoughts following our telecon: >>> >>>Jacek Becla wrote: >>> >>> >>> >>>>Keywords: DataAccWG >>>> >>>>Hi, >>>> >>>>I just had a quick word with Kem, here is the summary plus >>>>some estimates/comments I added. >>>> >>>> >>>>1) What is the expected size of pre-cached archived data >>>>at the base camp? >>>>We are expecting to have roughly 100TB of image data, >>>>corresponding catalog data will probably be ~10% of that >>>>So it is ~10TB (upper bound). >>>> >>>>2) How much of that data will be needed on average per image? >>>>~5GB (upper bound) >>>> >>>> >>>> >>>I think we need a combination of deep object data, which does not have >>>time history (it is constructed from deep stacks of images over a long >>>time span), and shallower object data which does have time history. For >>>the deep object data, if we assume 50 galaxies per arcmin**2, we have >>>about 2e6 objects per image. We need to work on the object data >>>definition, but let's suppose that for each filter we keep three >>>different kinds of magnitudes and their associated errors. We will >>>have some kind of classification, and a variety of data quality flags. >>>There will also be detailed shape information, but we likely don't need >>>this at base, so I ignore that here. If each of those mags/errors is a >>>32 bit float, the flags take 16 bits, and the classification 8 bits, and >>>we have 5 bands (U will not go deep), then we need 2e6*5 filters*(3 >>>mags*2 quantities/mag * 4 bytes/quantity + 3 bytes for flags) = 2.7 GB. >>> >>>The time dependent data will basically be limited to stars at roughly >>>25th in V. The density on the sky is highly variable, but an average >>>value of about 2 per arcmin**2 is probably not too far off. This gives >>>us roughly 1e5 stars per image on average. I think we will need summary >>>information for each star, plus perhaps the most recent ten measurements >>>(in whichever bands we have them). Again, we suffer from not having >>>defined our object data well. But suppose we summarize the star with >>>10 floats per band, and then have for each time point 2 floats plus 2 >>>bytes of quality flags. Then for each star we need 5*10*4 + 10*(2*4 + >>>2) = 300 bytes. The total for stars is then 0.07 GB. >>> >>>So, given the assumptions about what we need, I certainly think that >>>Kem's number is in the right ballpark. However, I wonder if we really >>>need to prestage all this data for every image. After all, we are >>>operating on difference images at base, and even granted that the >>>detection are supplemented by some list of objects to always measure, we >>>will be needing object information for only a tiny fraction of the stars >>>in the image - perhaps a few percent. So, we need to fetch info for a >>>few * 1e3 objects from the total of 2e6 total (mostly deep) objects for >>>which we have data. Does this change our strategy? >>> >>>Tim >>>_______________________________________________ >>>LSST-data mailing list >>>[email protected] >>>http://www.lsstmail.org/mailman/listinfo/lsst-data >>> >>> >>> >> >>_______________________________________________ >>LSST-data mailing list >>[email protected] >>http://www.lsstmail.org/mailman/listinfo/lsst-data >> >> > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
