Hi Arun
I believe the load related to locating image files through database
will be much smaller than the load related to multiple joins on
multi-billion-row tables, that is why we are not fervently working
on this yet. Can someone remind me how many files we will have?
2000 exposures per night x 300 nights per year = 0.5 million files?
That is realistic from file system point of view, and easy
from database point of view
2000 exposures per night x 300 nights per year x 201 files per exposure
= 100 million files?
That is not practical from file system point of view, and still
quite easy from database point of view
cheers,
Jacek
Arun Jagatheesan wrote:
I have a concern here on how we are proceeding wrto our IO estimates and
storage estimates....
This concern arises from what I assume as a DGMS specific queries. Currently
our database disk io estimates have considered only LSST schema in the
database.
If we want astronomers to discover 'files' using a DGMS, the database would
have DGMS schema also. We have not considered this in our current work. The
case 0, case 2 and case 3 discussed below all assume the use of DGMS. Should
we postpone the discussion on what are the additional database requirements
for DGMS schema (apart from LSST schema). Or on the other hand should we
discuss everything together? It must be noted that if we follow case 2 or
case 3, the LSST schema would require some changes or additional
relationships (tables).
Cheers,
Arun
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Sent: Friday, June 30, 2006 12:54 PM
To: [email protected]; [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: RE: [LSST-data] File or image retrieval and access patterns?
Thanks for the extra information.
If we think of 'LSST user' as more than just individuals, but
also research teams, or institutions, or other data
aggregators, then yes, it is possible that your sci-fi
scenario (Case 3) could be very useful, since QoS measures
will be important for planning purposes.
However, I believe that even individual astronomers should be
warned if their query exceeds some size limit or
time-to-delivery criteria. Even if they do not ask for QoS
metadata or use it to set limits on their query, nevertheless
the DGMS should provide some kind of notification or warning
or system administrator-imposed limits on potentially
enormous ad hoc queries.
For example, one of Jacek's sample queries produces several
Petabytes of intermediate results, to be sorted and filtered
in order to produce the final end-user result --- the
end-user results might be very small, and hence the end-user
may have no clue that their query has consumed every byte of
storage, every cycle of CPU, and every GBs of bandwidth to
generate their little answer. Some QoS measures should be
available to intervene in cases like this.
- Kirk
Date: Fri, 30 Jun 2006 11:09:12 -0700
From: Arun Jagatheesan <[EMAIL PROTECTED]>
Subject: RE: [LSST-data] File or image retrieval and access
patterns?
To: [EMAIL PROTECTED], "'LSST Data Management'"
<[email protected]>
In the upcoming versions of SRB DGMS, the metadata (LSST
schema) could
be part of the SRB system metadata it self. We call it
extended MCAT.
This allows queries or file-based requests to have joins on both
file-system metadata and domain-specific metadata.
Expanding more on this DGMS perspective, I get an additional
possibility. I am thinking loud here...
Case 0: LSST DB maintains LSST schema. SRB MCAT maintains the DGMS
schema of distributed files in Data Access center and other centers
that might join the LSST data grid collaboration. This will
mean the
middleware application will have to query both the schemas to have
some infrastructure-dependent domain queries that need
files as output.
Case 2: LSST schema and DGMS Schema could be maintained in a single
SRB DGMS. Joins are possible during the queries (at this
time it's not
mature as we don't have much production users willing to be
the guinea pigs).
Case 3: This is out of the scope of LSST design now as its
sci-fi for 2006.
It assumes LSST will have multiple data access centers
delivering some
part of the files or the database. LSST schema and DGMS
schema are on
distributed databases. SRB can already pick up the closest
replica and
other things based on some heuristics. But we can get more
aggressive
here. Astronomers could specify infrastructure details like
the "time
to wait" and "size of data" they are expecting. Based on this QoS
information, the queries could take advantage of the DGMS data can
gather (get) more or less file-based data that might be
distributed at multiple storage systems.
Cheers,
Arun
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
[EMAIL PROTECTED]
Sent: Friday, June 30, 2006 10:21 AM
To: [email protected]
Subject: Re: [LSST-data] File or image retrieval and
access patterns?
Yes, yes, yes. Though I don't see the clear distinction between
Case 2 and Case 3, unless you mean that the SRB metadata
are used in
Case 3 and the normal LSST DB metadata are used in Case 2.
- Kirk
Date: Fri, 30 Jun 2006 10:04:39 -0700
From: Arun Jagatheesan <[EMAIL PROTECTED]>
Subject: [LSST-data] File or image retrieval and access
patterns?
To: "'LSST Data Management'" <[email protected]>
This is a question for which I have been trying to find an
answer. I
would appreciate any friendly enlightenment about the type
of queries
astronomers might have.
Will the queries by astronomers be purely database-oriented
or will it
also lead to retrieval of image files from a data grid type
of system?
Case 1: Query the database using only astronomy related
attributes to
get result sets only from database. This is purely RDBMS here -
traditional SQL type queries.
Case 2: Query the database using only astronomy related
attributes to
get result sets from the database and some files or images.
(its more
DGMS style queries here) Case 3: Query the database using both
astronomy related attributes and storage-related
attributes to get
result sets both from the database and image archive
(e.g.) Query on ra, dec etc., and get these metadata, In
addition get
me the associated files only iff they can be retrieved with
certain QoS or with a delivery time of these much
hours. Or... I
don't want 5TB of images coming out of this query but I
am ok with 100GB.
Thanks in advance,
Arun
~~~~~~~~~
Luck is what happens when preparation meets opportunity.
Arun swaran Jagatheesan
http://www.sdsc.edu/~arun/
San Diego Supercomputer Center.
(858)822.5452
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
Jacek Becla
Sent: Thursday, June 29, 2006 9:24 PM
To: LSST Data Management
Subject: [LSST-data] Database telecon Fri Jun 30
Keywords: DataAccWG
Hi all,
We will have a Database telecon this Friday at 11:00 AM PDT.
I'd like to go over the latest disk io numbers.
Here is the latest version:
http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.do
c
http://www.slac.stanford.edu/~becla/tmp/lsst_diskIO_estimates_v04.xl
s
Shortly speaking the numbers looks very reasonable for
everything except two temporal queries, and both have
the same
problem (see
below)
Phone number: 866 330 1200
Pass code: 300 2363
Jacek
======================
Select time series data for given cone:
SELECT o.objectId, o.ra, o.dec, i.MJD, d.*
FROM Object o, CCDImage i, DIASource d
WHERE o.ra BETWEEN ra1 + cos(decl1)*(0.5)
AND ra1 - cos(decl1)*(0.5)
AND o.decl between decl1 +0.5 and decl1 - 0.5
AND o.objectId = d.objectId
AND d.ccdImageId = i.ccdImageId ORDER BY
o.objectId, i.MJD
ASC
Object table: 10 million rows (one partition)
DIASource table: 45 billion rows CCDImage table:
135 million
rows
At the moment, if I assume a modest 1% selectivity for
ra,dec that
gives 10K rows from Object table.
Join with DIASource (5 DIASources per star) gives 50K
rows, then
join with CCDImage (assuming 10K DIASources per
CCDImage) gives 50*10^12 rows. That is over a petabyte, which
now needs to be sorted, then for each row we need to
fetch full
DIASource row....
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
--------------------------------------------------------------
---------------
Dr. Kirk D. Borne
NASA Goddard Space Flight Center, SSDOO Program Manager,
QSS Group
Inc.
and George Mason University, Associate Research
Professor, College
of Science <mailto:[EMAIL PROTECTED]> Tel.
+1-301-286-0696 Fax: 301-286-1771
Staff page: http://rings.gsfc.nasa.gov/~borne/
US Virtual Observatory: http://www.us-vo.org/ Large
Synoptic Survey
Telescope: http://www.lssto.org/
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
--------------------------------------------------------------
---------------
Dr. Kirk D. Borne
NASA Goddard Space Flight Center, SSDOO Program Manager, QSS
Group Inc.
and George Mason University, Associate Research Professor,
College of Science <mailto:[EMAIL PROTECTED]> Tel.
+1-301-286-0696 Fax: 301-286-1771
Staff page: http://rings.gsfc.nasa.gov/~borne/
US Virtual Observatory: http://www.us-vo.org/ Large Synoptic
Survey Telescope: http://www.lssto.org/
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
