Keywords: DataAccWG Hi,
Maria, Ani, Serge, and I had a small discussion about object table partitioning, here is a short summary.
Two approaches under consideration: 1) one partition per zone (a zone = "horizontal stripe"), clustered index on RA 2) chop zones into smaller "chunks", 1 chunk = 1 partition Number 1 is easier to partition. Number 2 has potential to be less I/O intensive. Number 1 requires more data to tests. We will test and compare both approaches. How to test? - we should test with 3 full stripes. - height of each stripe = height of FOV - ok to test with smaller object density per sq deg (comparing to real lsst numbers) If we use lsst numbers: - FOV = 3.3 x 3.3 sq deg - average number of objects per FOV = 4 million - max number of objects per FOV = 10 million - 2.1 K row size in Object table we are getting: - 360 / 3.3 = 110 FOVs per stripe, so 330 FOVs in 3 stripes - 330 x 4 million objects x 2.1 K per object = 2.6 TB - 330 x 10 million objects x 2.1 K per object = 6.5 TB - plus indexes, swap space... [my comment after the meeting: we should probably start with ~1 million objects per FOV, which would be 650 GB of data] Perhaps will initially use simulator provided by Sergei (LLNL) - need to talk to Sergei - need to update prototype schema Could use small lsst-db testbed at slac (2 machines, 1TB on each) - will get accounts for Serge and Ani, Maria already has one Will continue this discussion this Wednesday (db telecon) and Friday (3 hour DataAccWG telecon) Jacek _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
