Keywords: DataAccWG Hello,
I made rather significant change in the estimates: we were forgetting that Deep Detection will run _for each band_ (which means many more Sources...). Building this into the model resulted in: - number of Sources increased ~300% - db size increased ~130% to 378 TB after the first year - disk io: number of disks increased ~x2 to 3,922 - size of RAM needed per processing center shot up from 6 to 19 TB, without this memory we are talking about 1 billion (yes, Billion) disks :=) The updated docs are in Docushare, Collection-413. BTW, all numbers were discussed with Tim and Kem and approved by Tim. Jacek Jacek Becla wrote:
Keywords: DataAccWG Hello, I refreshed the db storage and disk io estimate documents in Docushare, I believe these versions are very close to the final now. The changes include: storage (size up by 1%): - added 1% var galaxies disk io (number of disks up by 14%, 1,834 now): - added reverse estimates (how many queries we can handle for given number of disks) - fixed issue related to clustering galaxies (now that we have much fewer galaxies than stars the model should reflect worse hit ratio for galaxies, and it was not) Jacek Jacek Becla wrote:Keywords: DataAccWG Hello, I published in docushare the latest version of database storage estimates and disk io estimates (Collection-413) It includes latest numbers/assumptions provided by Tim, and the changes discussed/approved at the database telecon (Jul 12). In summary, we now expect - to have 164 TB of data after the first year (down from 600) - to need 1.6 K disks for the assumed query load (down from 4 K) Note that: 1) there is still a small debate over the galaxy numbers for DR20 ("should be more like 18 billion"), and the star numbers ("should be the 10 billion from the r-band counts"), so further updates are likely. 2) We are still working on details regarding which table should be used by which query (DIASource vs Source). This can change disk io (reduce it - now I am using Source) 3) I introduced small data duplication by extracting 6 rows from the Source table into a separate table in order to reduce disk io for the near-neighbor query by 85% (which reduced overall disk requirement by half). That should be ok given we asked for 150% of data size for this sort of "persistent overheads" (and are now "using" 8 TB out of 53 TB in DR1, or 15%) One worry that I have is that the estimates are now for DR1 and in DR1 the Source table is "tiny" (if one can call 40-billion-row table "tiny") comparing to what we will have to deal with later (for example in release 5 it is 10x bigger). The used DR number is selected based on number of objects in Object Catalog: we need 10 billion and DR1 has over 22 already. Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data_______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
