Keywords: DataAccWG
Hello,
I published in docushare the latest version of database
storage estimates and disk io estimates (Collection-413)
It includes latest numbers/assumptions provided by Tim,
and the changes discussed/approved at the database
telecon (Jul 12).
In summary, we now expect
- to have 164 TB of data after the first year (down from 600)
- to need 1.6 K disks for the assumed query load (down from 4 K)
Note that:
1) there is still a small debate over the galaxy numbers
for DR20 ("should be more like 18 billion"), and the star
numbers ("should be the 10 billion from the r-band counts"),
so further updates are likely.
2) We are still working on details regarding which table
should be used by which query (DIASource vs Source). This
can change disk io (reduce it - now I am using Source)
3) I introduced small data duplication by extracting 6 rows
from the Source table into a separate table in order to reduce
disk io for the near-neighbor query by 85% (which reduced
overall disk requirement by half). That should be ok given
we asked for 150% of data size for this sort of
"persistent overheads" (and are now "using" 8 TB out of 53 TB
in DR1, or 15%)
One worry that I have is that the estimates are now for
DR1 and in DR1 the Source table is "tiny" (if one can call
40-billion-row table "tiny") comparing to what we will
have to deal with later (for example in release 5
it is 10x bigger). The used DR number is selected based on
number of objects in Object Catalog: we need 10 billion and
DR1 has over 22 already.
Jacek
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
