Keywords: DataAccWG
Attendees:
Jeff Kantor
Russell Owen
Ani Thakar
Kirk Borne
Tim Axelrod
Ray Plante
Kem Cook
Jacek Becla
database storage estimates
- latest version uses input from Zeljko, database size
increased by ~50%
- now it is 600 TB data after the first year
- Tim: number of sources seems too large, will check the
spreadsheet in more details and start email discussion
Database disk io estimates
--> use the number of objects based on recent storage
estimate spreadsheet, not the 10 billion
- the number depends on what Tim finds, in the current
version we have 50 billion objects in DR1, so #disks
needed is likely to go up
- point raised by Jim Gray by email: with 300 queries and
4K disks disk bandwidth is much more than network bandwidth
(300 queries, 10 queries per sec, size of returned result
0.5 GB --> 15 GBytes/sec needed to ship results to users)
- so need to look at network side
- simultaneosity of low volume queries
- spreadsheet now assumes all 300 queries run truly
simultaneously
- sloan: about 20 low-volume users active at any given time
--> assume 50 truly simultaneous low volume users.
It is conservative. Rationale: more than 50 users will be on
at any given time, but they don't constantly run queries,
they need to wait for result, than think before submitting
new query (think time > wait time)
Supporting older releases
- SDSS: - a lot of non-science people use old releases, likely
because unaware of existence of new releases, or have url
to old release hardcoded somewhere
- assume that we will steer people to newer releases
--> assume 10% of all low volume users will use old releases
- guess based on sloan numbers and assumption that we will
steer users to new releases
- so 10% of low volume queries in our disk io model will use
old releases (and likely old releases will be on slower disks),
don't fine-tune the model to adjust for that
- don't put the numbers for old releases in the database
size estimates because the estimates for archive center
infrastructure already have some disks allocated for this purpose
- just document this assumption
Does the existence of two large centers (processing and archive)
double performance of queries?
- no, the load will be split between these two, likely
archive: high volume, data access: low volume
- BTW, that means we need x2 RAM (to cache indexes in both places)
Don't worry about CPU load yet, that will be next step
once we are done with disks
Super-high-volume query
- a chain of high volume queries
- except, result from one query is fed to the next query, so
the returned data set is not proportionally bigger
--> to simulate super-high-volume query, adjust numbers for
high volume queries:
- number of queries: 10 --> 20
- returned result set: leave at 6GB/query
- relax response time: 10 min --> 1 hour
Because of the network bandwidth limits, smaller number of low volume
queries and longer response time for high volume queries, we
can assume larger disks and slower response time,
--> use 8 msec seek time and assume large disks
Should we use Source instead of DIASource tables in some of our queries?
- yes, this was a mistake, e.g. retrieve light curve within a cone
needs Source, not DIASource
- Source is ~97% bigger than DIASource, so this will drive # disks up
--> now we really have to assume partitioning Source table
- this will drive #disks up, not sure how much
Apply changes discussed today, target: Wednesday next week,
then freeze and "look at our baseline configuration and work
backwards to the number of users that we can effectively support
with the baseline."
Terminology
- sources, vs detections vs visits
- deep db vs object catalog vs coadded catalog
- time db vs source catalog
- suggestion: use terminology used in domain model
- Tim will look at the domain model and send email about that
No telecon this Friday unless I run into issues that need discussion.
thanks,
Jacek
Jacek Becla wrote:
Keywords: DataAccWG
Hi all,
We will have a Database telecon tomorrow (Wed) at 11:00 AM PDT.
Agenda:
- disk space for staging older releases
- update on disk io and storage estimates
Phone number: 866 330 1200
Pass code: 300 2363
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
