Hi Jacek, Thanks for clarifying the expected file count. I had assumed users might be interested in looking at all the files (cumulative growth each year) in the archive rather than just for one release - that was the reason I was concerned if we might need to take care of this during our estimation for db.
If we used LSST schema and DGMS schema independent of each other (Case 0), the DB size of DGMS in this case will be very small as you mention. Currently one of our production DGMS instance uses around 100 GB for 50 million files. I hope this could even go into the some memory by 2012. On the other hand, if users wanted to be more aggressive (case 2 or 3) - with faster ways to discover "images or files" using an extensive domain-specific meta data attributes, the DB size could increase. Again if the file count is 'less' and we are required to go through only one release it should not matter much and we can safely ignore this. Cheers, Arun > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Jacek Becla > Sent: Wednesday, July 05, 2006 11:20 AM > To: LSST Data Management > Subject: Re: [LSST-data] File or image retrieval and access patterns? > > Hi Arun > > I believe the load related to locating image files through > database will be much smaller than the load related to > multiple joins on multi-billion-row tables, that is why we > are not fervently working on this yet. Can someone remind me > how many files we will have? > > 2000 exposures per night x 300 nights per year = 0.5 million files? > That is realistic from file system point of view, and easy > from database point of view > > 2000 exposures per night x 300 nights per year x 201 files > per exposure = 100 million files? > That is not practical from file system point of view, and > still quite easy from database point of view > > cheers, > Jacek > > > > > Arun Jagatheesan wrote: > > I have a concern here on how we are proceeding wrto our IO > estimates > > and storage estimates.... > > > > This concern arises from what I assume as a DGMS specific queries. > > Currently our database disk io estimates have considered only LSST > > schema in the database. > > > > If we want astronomers to discover 'files' using a DGMS, > the database > > would have DGMS schema also. We have not considered this in our > > current work. The case 0, case 2 and case 3 discussed below > all assume > > the use of DGMS. Should we postpone the discussion on what are the > > additional database requirements for DGMS schema (apart from LSST > > schema). Or on the other hand should we discuss everything > together? > > It must be noted that if we follow case 2 or case 3, the > LSST schema > > would require some changes or additional relationships (tables). > > > > Cheers, > > Arun > > > > > > > >>-----Original Message----- > >>From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > >>Sent: Friday, June 30, 2006 12:54 PM > >>To: [email protected]; [EMAIL PROTECTED] > >>Cc: [EMAIL PROTECTED] > >>Subject: RE: [LSST-data] File or image retrieval and access > patterns? > >> > >>Thanks for the extra information. > >> > >>If we think of 'LSST user' as more than just individuals, but also > >>research teams, or institutions, or other data aggregators, > then yes, > >>it is possible that your sci-fi scenario (Case 3) could be very > >>useful, since QoS measures will be important for planning purposes. > >> > >>However, I believe that even individual astronomers should > be warned > >>if their query exceeds some size limit or time-to-delivery > criteria. > >>Even if they do not ask for QoS metadata or use it to set limits on > >>their query, nevertheless the DGMS should provide some kind of > >>notification or warning or system administrator-imposed limits on > >>potentially enormous ad hoc queries. > >> > >>For example, one of Jacek's sample queries produces several > Petabytes > >>of intermediate results, to be sorted and filtered in order > to produce > >>the final end-user result --- the end-user results might be very > >>small, and hence the end-user may have no clue that their query has > >>consumed every byte of storage, every cycle of CPU, and > every GBs of > >>bandwidth to generate their little answer. Some QoS > measures should > >>be available to intervene in cases like this. > >> > >>- Kirk > >> > >> > >> > >>>Date: Fri, 30 Jun 2006 11:09:12 -0700 > >>>From: Arun Jagatheesan <[EMAIL PROTECTED]> > >>>Subject: RE: [LSST-data] File or image retrieval and access > >> > >>patterns? > >> > >>>To: [EMAIL PROTECTED], "'LSST Data Management'" > >> > >><[email protected]> > >> > >>>In the upcoming versions of SRB DGMS, the metadata (LSST > >> > >>schema) could > >> > >>>be part of the SRB system metadata it self. We call it > >> > >>extended MCAT. > >> > >>>This allows queries or file-based requests to have joins on both > >>>file-system metadata and domain-specific metadata. > >>> > >>>Expanding more on this DGMS perspective, I get an additional > >>>possibility. I am thinking loud here... > >>> > >>>Case 0: LSST DB maintains LSST schema. SRB MCAT maintains the DGMS > >>>schema of distributed files in Data Access center and > other centers > >>>that might join the LSST data grid collaboration. This will > >> > >>mean the > >> > >>>middleware application will have to query both the schemas to have > >>>some infrastructure-dependent domain queries that need > >> > >>files as output. > >> > >>>Case 2: LSST schema and DGMS Schema could be maintained in > a single > >>>SRB DGMS. Joins are possible during the queries (at this > >> > >>time it's not > >> > >>>mature as we don't have much production users willing to be > >> > >>the guinea pigs). > >> > >>>Case 3: This is out of the scope of LSST design now as its > >> > >>sci-fi for 2006. > >> > >>>It assumes LSST will have multiple data access centers > >> > >>delivering some > >> > >>>part of the files or the database. LSST schema and DGMS > >> > >>schema are on > >> > >>>distributed databases. SRB can already pick up the closest > >> > >>replica and > >> > >>>other things based on some heuristics. But we can get more > >> > >>aggressive > >> > >>>here. Astronomers could specify infrastructure details like > >> > >>the "time > >> > >>>to wait" and "size of data" they are expecting. Based on this QoS > >>>information, the queries could take advantage of the DGMS data can > >>>gather (get) more or less file-based data that might be > >> > >>distributed at multiple storage systems. > >> > >>>Cheers, > >>>Arun > >>> > >>> > >>> > >>>>-----Original Message----- > >>>>From: [EMAIL PROTECTED] > >>>>[mailto:[EMAIL PROTECTED] On Behalf Of > >> > >>[EMAIL PROTECTED] > >> > >>>>Sent: Friday, June 30, 2006 10:21 AM > >>>>To: [email protected] > >>>>Subject: Re: [LSST-data] File or image retrieval and > >> > >>access patterns? > >> > >>>>Yes, yes, yes. Though I don't see the clear distinction between > >>>>Case 2 and Case 3, unless you mean that the SRB metadata > >> > >>are used in > >> > >>>>Case 3 and the normal LSST DB metadata are used in Case 2. > >>>> > >>>>- Kirk > >>>> > >>>> > >>>>>Date: Fri, 30 Jun 2006 10:04:39 -0700 > >>>>>From: Arun Jagatheesan <[EMAIL PROTECTED]> > >>>>>Subject: [LSST-data] File or image retrieval and access > >> > >>patterns? > >> > >>>>>To: "'LSST Data Management'" <[email protected]> > >>>>> > >>>>>This is a question for which I have been trying to find an > >>>> > >>>>answer. I > >>>> > >>>>>would appreciate any friendly enlightenment about the type > >>>> > >>>>of queries > >>>> > >>>>>astronomers might have. > >>>>> > >>>>>Will the queries by astronomers be purely database-oriented > >>>> > >>>>or will it > >>>> > >>>>>also lead to retrieval of image files from a data grid type > >>>> > >>>>of system? > >>>> > >>>>>Case 1: Query the database using only astronomy related > >>>> > >>>>attributes to > >>>> > >>>>>get result sets only from database. This is purely RDBMS here - > >>>>>traditional SQL type queries. > >>>>>Case 2: Query the database using only astronomy related > >>>> > >>>>attributes to > >>>> > >>>>>get result sets from the database and some files or images. > >>>> > >>>>(its more > >>>> > >>>>>DGMS style queries here) Case 3: Query the database using both > >>>>>astronomy related attributes and storage-related > >> > >>attributes to get > >> > >>>>>result sets both from the database and image archive > >>>>>(e.g.) Query on ra, dec etc., and get these metadata, In > >>>> > >>>>addition get > >>>> > >>>>>me the associated files only iff they can be retrieved > with certain > >>>>>QoS or with a delivery time of these much > >> > >>hours. Or... I > >> > >>>>>don't want 5TB of images coming out of this query but I > >> > >>am ok with 100GB. > >> > >>>>>Thanks in advance, > >>>>>Arun > >>>>>~~~~~~~~~ > >>>>>Luck is what happens when preparation meets opportunity. > >>>>> > >>>>>Arun swaran Jagatheesan > >>>>>http://www.sdsc.edu/~arun/ > >>>>>San Diego Supercomputer Center. > >>>>>(858)822.5452 > >>>>> > >>>>> > >>>>>>-----Original Message----- > >>>>>>From: [EMAIL PROTECTED] > >>>>>>[mailto:[EMAIL PROTECTED] On Behalf Of > >> > >>Jacek Becla > >> > >>>>>>Sent: Thursday, June 29, 2006 9:24 PM > >>>>>>To: LSST Data Management > >>>>>>Subject: [LSST-data] Database telecon Fri Jun 30 > >>>>>> > >>>>>>Keywords: DataAccWG > >>>>>> > >>>>>>Hi all, > >>>>>> > >>>>>>We will have a Database telecon this Friday at 11:00 AM PDT. > >>>>>> > >>>>>>I'd like to go over the latest disk io numbers. > >>>>>> > >>>>>>Here is the latest version: > >>>>>> > >>>>>> > >>>> > >>http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.do > >> > >>>>>>c > >>>>>> > >>>> > >>http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.xl > >> > >>>>>>s > >>>>>> > >>>>>> > >>>>>>Shortly speaking the numbers looks very reasonable for > everything > >>>>>>except two temporal queries, and both have > >> > >>the same > >> > >>>>>>problem (see > >>>>>>below) > >>>>>> > >>>>>>Phone number: 866 330 1200 > >>>>>>Pass code: 300 2363 > >>>>>> > >>>>>>Jacek > >>>>>> > >>>>>>====================== > >>>>>> > >>>>>>Select time series data for given cone: > >>>>>> > >>>>>>SELECT o.objectId, o.ra, o.dec, i.MJD, d.* > >>>>>>FROM Object o, CCDImage i, DIASource d > >>>>>>WHERE o.ra BETWEEN ra1 + cos(decl1)*(0.5) > >>>>>> AND ra1 - cos(decl1)*(0.5) > >>>>>> AND o.decl between decl1 +0.5 and decl1 - 0.5 > >>>>>> AND o.objectId = d.objectId > >>>>>> AND d.ccdImageId = i.ccdImageId ORDER BY > >> > >>o.objectId, i.MJD > >> > >>>>>>ASC > >>>>>> > >>>>>> > >>>>>>Object table: 10 million rows (one partition) > >>>>>>DIASource table: 45 billion rows CCDImage table: > >> > >>135 million > >> > >>>>>>rows > >>>>>> > >>>>>> > >>>>>>At the moment, if I assume a modest 1% selectivity for > >>>> > >>>>ra,dec that > >>>> > >>>>>>gives 10K rows from Object table. > >>>>>> > >>>>>>Join with DIASource (5 DIASources per star) gives 50K > >> > >>rows, then > >> > >>>>>>join with CCDImage (assuming 10K DIASources per > >>>>>>CCDImage) gives 50*10^12 rows. That is over a petabyte, > which now > >>>>>>needs to be sorted, then for each row we need to > >> > >>fetch full > >> > >>>>>>DIASource row.... > >>>>>> > >>>>>> > >>>>>>_______________________________________________ > >>>>>>LSST-data mailing list > >>>>>>[email protected] > >>>>>>http://www.lsstmail.org/mailman/listinfo/lsst-data > >>>>>> > >>>>> > >>>>>_______________________________________________ > >>>>>LSST-data mailing list > >>>>>[email protected] > >>>>>http://www.lsstmail.org/mailman/listinfo/lsst-data > >>>> > >>>>-------------------------------------------------------------- > >>>>--------------- > >>>>Dr. Kirk D. Borne > >>>>NASA Goddard Space Flight Center, SSDOO Program Manager, > >> > >>QSS Group > >> > >>>>Inc. > >>>>and George Mason University, Associate Research > >> > >>Professor, College > >> > >>>>of Science <mailto:[EMAIL PROTECTED]> Tel. > >>>>+1-301-286-0696 Fax: 301-286-1771 > >>>>Staff page: http://rings.gsfc.nasa.gov/~borne/ > >>>>US Virtual Observatory: http://www.us-vo.org/ Large > >> > >>Synoptic Survey > >> > >>>>Telescope: http://www.lssto.org/ > >>>> > >>>>_______________________________________________ > >>>>LSST-data mailing list > >>>>[email protected] > >>>>http://www.lsstmail.org/mailman/listinfo/lsst-data > >>>> > >> > >>-------------------------------------------------------------- > >>--------------- > >>Dr. Kirk D. Borne > >>NASA Goddard Space Flight Center, SSDOO Program Manager, QSS Group > >>Inc. > >>and George Mason University, Associate Research Professor, > College of > >>Science <mailto:[EMAIL PROTECTED]> Tel. > >>+1-301-286-0696 Fax: 301-286-1771 > >>Staff page: http://rings.gsfc.nasa.gov/~borne/ > >>US Virtual Observatory: http://www.us-vo.org/ Large > Synoptic Survey > >>Telescope: http://www.lssto.org/ > >> > > > > > > _______________________________________________ > > LSST-data mailing list > > [email protected] > > http://www.lsstmail.org/mailman/listinfo/lsst-data > > _______________________________________________ > LSST-data mailing list > [email protected] > http://www.lsstmail.org/mailman/listinfo/lsst-data > _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
