Yes, Mark... -----Original Message----- From: Mark Nelson [mailto:[email protected]] Sent: Wednesday, October 08, 2014 6:08 PM To: Somnath Roy; Duan, Jiangang; Andreas Bluemle; [email protected] Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Somnath, Was this with HT enabled/disabled on both the cluster and the RBD nodes? Mark On 10/08/2014 07:50 PM, Somnath Roy wrote: > Hi Jiangang, > I managed to get some data for you but it's for a 3 node cluster. I will try > to get data for single node as well. > > Test config: > ------------- > > Cluster and rbd node config: > ---------------------------------- > "2x E5-2680 10C 2.8GHz 25M > 8x 16GB RDIMM, dual rank x4 (128GB) > Mellanox MT27500 40 Gigabit Ethernet > LSI 9207 SAS HBA" > > 8 X 800 GB SSDs (Optimus Eco) per cluster node > > 3 cluster nodes + 3 rbd nodes > > Total storage ~ 19 TB > > We have total 24 OSDs running , each node has 8 OSDs/SSD > > Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is > ~230G. > > We have tried on 64K_RR_QD64 workload here. > > HT_ENABLE > -------------- > > IOPS : 112500 > Throughput (MB/S): 7012 > Avg Resp.Time (m.sec): 17 > Max Resp.Time (m.sec): 3184 > > HT_DISABLE > -------------- > > IOPS : 120864 > Throughput (MB/S): 7530 > Avg Resp.Time (m.sec): 11 > Max Resp.Time (m.sec): 1056 > > > So, ~7% iop increase but response time decrease is ~35% which is real good. > > Thanks & Regards > Somnath > > -----Original Message----- > From: Duan, Jiangang [mailto:[email protected]] > Sent: Wednesday, October 08, 2014 1:03 PM > To: Somnath Roy; Andreas Bluemle; [email protected] > Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot > params > > Sound good. Thanks. -jiangang > > -----Original Message----- > From: Somnath Roy [mailto:[email protected]] > Sent: Wednesday, October 08, 2014 10:53 AM > To: Duan, Jiangang; Andreas Bluemle; [email protected] > Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot > params > > Hi Jiangang, > Give me a day or two, I will gather all the data and share with community. > > Thanks & Regards > Somnath > > -----Original Message----- > From: Duan, Jiangang [mailto:[email protected]] > Sent: Wednesday, October 08, 2014 10:47 AM > To: Somnath Roy; Andreas Bluemle; [email protected] > Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot > params > > Can you guys share the w/ HT and w/o HT data? I want to take a look at that > to understand why. > > -jiangang > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Somnath Roy > Sent: Wednesday, October 08, 2014 10:38 AM > To: Andreas Bluemle; [email protected] > Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot > params > > Thanks Andres for sharing this. I will try those out. > BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like > 'cpufreq'.. > > root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/ > cache/ crash_notes driver/ microcode/ > online subsystem/ topology/ > cpuidle/ crash_notes_size firmware_node/ node0/ > power/ thermal_throttle/ uevent > > I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz. > > Regards > Somnath > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Andreas Bluemle > Sent: Wednesday, October 08, 2014 9:33 AM > To: [email protected] > Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot > params > > Hi, > > as mentioned during today's meeting, here are the kernel boot parameters > which I found to provide the basis for good performance results: > > processor.max_cstate=0 > intel_idle.max_cstate=0 > > I understand these to basically turn off any power saving modes of the CPU; > the CPU's we are using are like > Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz > Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz > > At the BIOS level, we > - turn off Hyperthraeding > - turn off Turbo mode (in order ot not leave the specifications) > - turn on frequency floor override > > We also assert that > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor > is set to "performance" > > Using above we see a constant frequency at the maximum level allowed by the > CPU (except Turbo mode). > > > Best Regards > > Andreas Bluemle > > > > > > > On Wed, 8 Oct 2014 02:51:21 +0200 > Mark Nelson <[email protected]> wrote: > >> Hi All, >> >> Just a remind that the weekly performance meeting is on Wednesdays at >> 8AM PST. Same bat time, same bat channel! >> >> Etherpad URL: >> http://pad.ceph.com/p/performance_weekly >> >> To join the Meeting: >> https://bluejeans.com/268261044 >> >> To join via Browser: >> https://bluejeans.com/268261044/browser >> >> To join with Lync: >> https://bluejeans.com/268261044/lync >> >> >> To join via Room System: >> Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID: >> 268261044 >> >> To join via Phone: >> 1) Dial: >> +1 408 740 7256 >> +1 888 240 2560(US Toll Free) >> +1 408 317 9253(Alternate Number) >> (see all numbers - http://bluejeans.com/numbers) >> 2) Enter Conference ID: 268261044 >> >> Mark >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to [email protected] More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> > > > > -- > Andreas Bluemle mailto:[email protected] > ITXperts GmbH http://www.itxperts.de > Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 > D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 > > Company details: http://www.itxperts.de/imprint.htm > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to [email protected] More majordomo > info at http://vger.kernel.org/majordomo-info.html > > ________________________________ > > PLEASE NOTE: The information contained in this electronic mail message is > intended only for the use of the designated recipient(s) named above. If the > reader of this message is not the intended recipient, you are hereby notified > that you have received this message in error and that any review, > dissemination, distribution, or copying of this message is strictly > prohibited. If you have received this communication in error, please notify > the sender by telephone or e-mail (as shown above) immediately and destroy > any and all copies of this message in your possession (whether hard copies or > electronically stored copies). > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to [email protected] More majordomo > info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to [email protected] More majordomo > info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
