For simple predicates, typical of SDSS, involving at most a few transcendental functions and lots of math on about 10 fields we can search at 1 M records/second/ GHz (that's about 1,000 instructions/record with a clocks/instruction (cpi) of about 1 (typical of decision support system (dss) workloads). (really simple predicates go 3x faster). GHz/chip is going to increase fast (due to multicore -- probably to 64 cores by 2014 (the new event horizon). And this search stuff is embarrassingly parallel. So we get about 128 GHz/chip in that time frame if you can use 64 threads and so 128 M records/second/chip. If your records are 100 bytes (indices and vertical partitioning and compression are your friends). Then you have a 12GB/s/chip search engine with a one-chip cpu. That is about 60 disks / cpu chip (and a LOT of RAM chips and bandwidth per cpu).
A good book about IPS (instructions per second) = CPI * clocks/second is Patterson-Hennessey: http://www.amazon.com/gp/product/1558603298/103-4189199-8747028?v=glance &n=283155 But... 60 disks per cpu chip should give you a sense that you are IO BOUND not CPU bound. And ... Disks dominate the cost of your system. Indeed, most internet search guys "waste" cpus by having a mere 8 or 16 disks per cpu chip. So... To repeat.... Its the IO and memory bandwidth that is going to be your problem. NOT cpus, NOT disk space. Jim Gray Microsoft Research, Suite 1690, 455 Market, SF CA 94105, tel: 415 778 8222 fax: 425 706 7329 [EMAIL PROTECTED] http://research.Microsoft.com/~gray -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ray Plante Sent: Tuesday, July 25, 2006 9:49 AM To: LSST Data Management Subject: [LSST-data] calcuating flops [TechAssessWG] Keywords: TechAssessWG Hi folks, Here's how I estimated the flops needed for Jacek's CPU requirement estimate for supporting DB searching. This was based on lore gleaned from a Google search on the subject. I understand that the theoretical/peak flops of a cpu is usually 2 times the nominal clock rate, so... Psus = 2 * Scpu * Nproc * Rsus where Psus = the sustained performance in flops Scpu = the speed of the CPU used (1.8 GHz) Nproc = the number of processors needed (5329) Rsus = reality factor Psus/Ppeak; which I picked as 65% from an unscientific sampling of PC clusters. As I mentioned, I'm no expert on this, so your corrections are welcome. cheers, Ray _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
