Keywords: DataAccWG
Attendees: Tim, Ray, David, Arun, Kem, Maria, Jacek
Bottom-up db size estimates
===========================
How was the top-down estimate done?
- 10% of image size (~12 TB/night) --> ~1.2 TB/night
for catalog --> ~300 TB/year
Current baseline says: "0.4 - 4.5 TB/night for catalog"
Most recent top down estimate: ~0.04 TB/night, <15 TB/year
- x20 smaller than top down!
How believable the current bottom-up estimate is?
- detections: sizing a bit unknown, but likely ~2-3 x time db size
- facility database not counted, but it is small
- near neighbor data not counted, that is ~10% of total data size
- overheads not counted, but they were not counted in the top-down
estimate either
- so the numbers seem to well reflect the size of data that we
expect to store in the lsst database
- ...but they are very hard to believe (several of us think)
Very important to have a correct estimate:
- 300 vs 15 TB/year makes a huge difference how we
approach designing the database
- e.g. we could _easily_ keep all indexes in RAM for 15 TB
of data in 2012 (or even now...)
Action items:
- come up with better average numbers for time-dependent db
- Tim/Kem will review the whole bottom-up approach
- Tim: no time until end of next week
- Kem: will look this weekend
Hypothetical end user access workload
=====================================
Is the number of users realistic?
- "pulled out of the air" by Jeff and Tim
- the 200 is meant to reflect (mostly) general public access
- it seems pretty intense
Would it be useful to look at sdss numbers?
- data about sdss query load is available on the web
- e.g. in 2005: 8 million queries answered, that is 22/day,
new query every 4 sec
- sdss measures queries/day, not simultaneous users
- Ani knows more about sdss load
- difficult to scale lsst from Sloan numbers
--> let's do the exercise with the suggested numbers, we can do
another exercise later based on what we learn from this one
Assumptions we will be making:
- data type: mixture of spatial and temporal
- do not re-use queries: each query should be different
- catalog size: 10 billion objects
- because we assume that the super-high-volume query goes
through all objects
- based on current bottom-up estimates, that is ~DR4
DR1 --> 5 billion objects
DR2 --> 7
DR3 --> 8.5
DR4 --> 10
- so end of the year 2015
- assume 200 bytes per object (based on prototype schema)
- that is 2 TB
- Should we partition data or not?
- yes, tree-like structure is needed to do correlation
calculations (the super-high-volume is about correlations)
- assume one set of data, no replication
- assume clustering based on spatial attributes
need to know exact queries
- "bug" Kirk and Kem
- in the meantime, start with the 20 sdss queries, most of them
should map to lsst
need to better understand what hardware to assume
- Ray will provide some info: what NCSA assumed so far
for hardware in 2015
- things like disk sizes, disk speeds
need to decide what indexes we will have
- to do that, need to know schema and queries
might try to assume that we will have few separate
specialized catalogs for some smaller subsets of data
need to understand more the data contents
- e.g. sky coverage
- work with Kem, data from cadence simulator might help
-----
Most likely will continue this discussion next Wed during
Database telecon slot, and Friday (DataAccWG telecon slot).
Tim can not make Wed, Friday ok.
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi,
We will have a special DataAccWG telecon tomorrow at 11:00 AM PDT.
The main topic will be "hypothetical end user access workload",
see below for more details.
Phone number: 866 330 1200
passcode: 300 2363
Jacek
-------- Original Message --------
Subject: Hypothetical end user access workload
Date: Thu, 08 Jun 2006 10:01:50 -0700
From: Jeffrey P Kantor <[EMAIL PROTECTED]>
To: Ray Plante <[EMAIL PROTECTED]>, Jacek Becla
<[EMAIL PROTECTED]>, Arun Jagatheesan <[EMAIL PROTECTED]>
CC: Tim Axelrod <[EMAIL PROTECTED]>
Hello all,
Until we have a true model of user access load, I would like to price a
"unit" of load to support the following numbers of SIMULTANEOUS users:
200 low-volume users querying against stored meta-data only over 10 million
objects in the object catalog or 1 degree^2 of the image archive,
retrieving
a 1 Gbyte dataset from a single cluster/partition, and receiving it at
100Mbps. Query response time should be less than 10 seconds.
10 high-volume users querying against stored and derived meta-data with
moderately complex conditions, over 1 billion objects in the object catalog
or the full FOV of the image archive, retrieving a 6 Gbyte dataset from
multiple clusters/partitions, and receiving it at 1Gbps. Query response
time should be less than 10 minutes.
1 super-high-volume user querying against stored and derived meta-data with
highly complex conditions and 3-point correlation, over 10 billion objects
in the object catalog, retrieving a 10 Gbyte dataset from all
clusters/partitions, and receiving it at 10Gbps. Query response time
should be less than 10 days.
Please make additional assumptions you need to and document them. Then
tell
me a representative set of file, database, and network servers that can
support this. Please do this as a team and have this ready for my
return on
6/23. Pose questions to Tim as necessary.
Thanks!
Jeff
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
