Hi all,
> Hello all, > > This fits very well into the approach I suggested (split > the sky into 5000 2GB-tables), the only change would be > adding an extra step that selects small number of objects definitely I'm missing something so got two questions? Where is the full set of objects? How do you select the small number of objects out of the big set of objects? Cheers, Maria > and loads them to memory, instead of blindly loading > one 2GB table and parts of few adjacent tables. This further > reduces required disk IO at the Base - very nice :-) > > Jacek > > > > > Kem Cook wrote: > > Hi, > > > > Got it. Staging the queries that way probably would be faster and more > > efficient. > > > > cheers, > > > > Kem > > > > > >>Hi Kem, > >> > >>I am in complete agreement! I guess I wasn't clear enough in what I > >>was proposing. I am not suggesting that we ignore the deep detection > >>data in alerting at base. What I *am* suggesting is that it may be > >>more efficient to query that data for the relatively small number of > >>detected transients rather than to blindly load it all into memory for > >>every frame. > >> > >>Tim > >> > >>Kem Cook wrote: > >> > >> > >>>Hi, > >>> > >>>So, we may not need all the static data, but I thought that it would be > >>>value added to be able to say that a new transient occured at the > >>>position > >>>of a known, but non-variable object (say a distant, barely resolved > >>>galaxy), or that a new transient appeared x arcsec offset from the > >>>nucleus of a rather well resolved galaxy. or that there was a transient > >>>at > >>>the position of a known M dwarf (perviously non-variable). > >>> > >>>Kem > >>> > >>> > >>> > >>> > >>>>Hi, > >>>> > >>>>A few thoughts following our telecon: > >>>> > >>>>Jacek Becla wrote: > >>>> > >>>> > >>>> > >>>> > >>>>>Keywords: DataAccWG > >>>>> > >>>>>Hi, > >>>>> > >>>>>I just had a quick word with Kem, here is the summary plus > >>>>>some estimates/comments I added. > >>>>> > >>>>> > >>>>>1) What is the expected size of pre-cached archived data > >>>>>at the base camp? > >>>>>We are expecting to have roughly 100TB of image data, > >>>>>corresponding catalog data will probably be ~10% of that > >>>>>So it is ~10TB (upper bound). > >>>>> > >>>>>2) How much of that data will be needed on average per image? > >>>>>~5GB (upper bound) > >>>>> > >>>>> > >>>>> > >>>> > >>>>I think we need a combination of deep object data, which does not have > >>>>time history (it is constructed from deep stacks of images over a long > >>>>time span), and shallower object data which does have time history. For > >>>>the deep object data, if we assume 50 galaxies per arcmin**2, we have > >>>>about 2e6 objects per image. We need to work on the object data > >>>>definition, but let's suppose that for each filter we keep three > >>>>different kinds of magnitudes and their associated errors. We will > >>>>have some kind of classification, and a variety of data quality flags. > >>>>There will also be detailed shape information, but we likely don't need > >>>>this at base, so I ignore that here. If each of those mags/errors is a > >>>>32 bit float, the flags take 16 bits, and the classification 8 bits, and > >>>>we have 5 bands (U will not go deep), then we need 2e6*5 filters*(3 > >>>>mags*2 quantities/mag * 4 bytes/quantity + 3 bytes for flags) = 2.7 GB. > >>>> > >>>>The time dependent data will basically be limited to stars at roughly > >>>>25th in V. The density on the sky is highly variable, but an average > >>>>value of about 2 per arcmin**2 is probably not too far off. This gives > >>>>us roughly 1e5 stars per image on average. I think we will need summary > >>>>information for each star, plus perhaps the most recent ten measurements > >>>>(in whichever bands we have them). Again, we suffer from not having > >>>>defined our object data well. But suppose we summarize the star with > >>>>10 floats per band, and then have for each time point 2 floats plus 2 > >>>>bytes of quality flags. Then for each star we need 5*10*4 + 10*(2*4 + > >>>>2) = 300 bytes. The total for stars is then 0.07 GB. > >>>> > >>>>So, given the assumptions about what we need, I certainly think that > >>>>Kem's number is in the right ballpark. However, I wonder if we really > >>>>need to prestage all this data for every image. After all, we are > >>>>operating on difference images at base, and even granted that the > >>>>detection are supplemented by some list of objects to always measure, we > >>>>will be needing object information for only a tiny fraction of the stars > >>>>in the image - perhaps a few percent. So, we need to fetch info for a > >>>>few * 1e3 objects from the total of 2e6 total (mostly deep) objects for > >>>>which we have data. Does this change our strategy? > >>>> > >>>>Tim > >>>>_______________________________________________ > >>>>LSST-data mailing list > >>>>[email protected] > >>>>http://www.lsstmail.org/mailman/listinfo/lsst-data > >>>> > >>>> > >>>> > >>> > >>>_______________________________________________ > >>>LSST-data mailing list > >>>[email protected] > >>>http://www.lsstmail.org/mailman/listinfo/lsst-data > >>> > >>> > >> > >>_______________________________________________ > >>LSST-data mailing list > >>[email protected] > >>http://www.lsstmail.org/mailman/listinfo/lsst-data > >> > > > > > > _______________________________________________ > > LSST-data mailing list > > [email protected] > > http://www.lsstmail.org/mailman/listinfo/lsst-data > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > -- ------------------------------------------------ Maria A. Nieto-Santisteban ([EMAIL PROTECTED]) Johns Hopkins University 3400 N. Charles St. Physics & Astronomy Department Baltimore, MD 21218 (USA) Tel: 1 410 516-7679 Fax: 1 410 516-5096 _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
