On 05/15/2018 02:51 PM, Blair Bethwaite wrote: > Sorry, bit late to get back to this... > > On Wed., 2 May 2018, 06:19 Nick Fisk, <n...@fisk.me.uk > <mailto:n...@fisk.me.uk>> wrote: > > 4.16 required? > > > Looks like it - thanks for pointing that out. > > Wido, I don't think you are doing anything wrong here, maybe this is a > bug... > > I've got RHEL7 + Broadwell based Ceph nodes here for which the same > tuning appears to be working fine: >
Odd indeed. Keep in mind that I indeed have the newer Intel Scalable CPUs with Ubuntu 16.04 and a 4.16 kernel. My main goal is the lowest possible latency with NVMe and for that you need higher clock speeds. (more down) > -bash-4.2$ lsb_release -a > LSB Version: :core-4.1-amd64:core-4.1-noarch > Distributor ID: RedHatEnterpriseServer > Description: Red Hat Enterprise Linux Server release 7.3 (Maipo) > Release: 7.3 > Codename: Maipo > > -bash-4.2$ lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 20 > On-line CPU(s) list: 0-19 > Thread(s) per core: 2 > Core(s) per socket: 10 > Socket(s): 1 > NUMA node(s): 1 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 79 > Model name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz > Stepping: 1 > CPU MHz: 2745.960 > BogoMIPS: 4399.83 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 25600K > NUMA node0 CPU(s): 0-19 > > -bash-4.2$ cat /proc/cmdline > BOOT_IMAGE=/vmlinuz-3.10.0-514.2.2.el7.x86_64 > root=/dev/mapper/vg00-LogVol00 ro nofb splash=quiet crashkernel=auto > rd.lvm.lv <http://rd.lvm.lv>=vg00/LogVol00 rd.lvm.lv > <http://rd.lvm.lv>=vg00/LogVol01 rhgb quiet LANG=en_US.UTF-8 > > -bash-4.2$ sudo cpupower frequency-info > analyzing CPU 0: > driver: intel_pstate > CPUs which run at the same hardware frequency: 0 > CPUs which need to have their frequency coordinated by software: 0 > maximum transition latency: Cannot determine or is not supported. > hardware limits: 1.20 GHz - 3.10 GHz > available cpufreq governors: performance powersave > current policy: frequency should be within 1.20 GHz and 3.10 GHz. > The governor "performance" may decide which speed to use > within this range. > current CPU frequency: 2.40 GHz (asserted by call to hardware) > boost state support: > Supported: yes > Active: yes > > -bash-4.2$ sudo cpupower -c 0-19 monitor > |Nehalem || Mperf || Idle_Stats > CPU | C3 | C6 | PC3 | PC6 || C0 | Cx | Freq || POLL | C1-B | > C1E- | C3-B | C6-B > 0| 0.00| 0.00| 0.00| 0.00|| 20.93| 79.07| 2398|| 1.00| 79.08| > 0.00| 0.00| 0.00 > 10| 0.00| 0.00| 0.00| 0.00|| 1.81| 98.19| 2398|| 0.00| 98.23| > 0.00| 0.00| 0.00 > 1| 0.00| 0.00| 0.00| 0.00|| 3.80| 96.20| 2398|| 2.10| 96.21| > 0.00| 0.00| 0.00 > 11| 0.00| 0.00| 0.00| 0.00|| 7.95| 92.05| 2398|| 7.59| 92.06| > 0.00| 0.00| 0.00 > 2| 0.00| 0.00| 0.00| 0.00|| 1.99| 98.01| 2398|| 0.00| 98.04| > 0.00| 0.00| 0.00 > 12| 0.00| 0.00| 0.00| 0.00|| 1.59| 98.41| 2398|| 0.64| 98.42| > 0.00| 0.00| 0.00 > 3| 0.00| 0.00| 0.00| 0.00|| 24.58| 75.42| 2398|| 0.00| 75.43| > 0.00| 0.00| 0.00 > 13| 0.00| 0.00| 0.00| 0.00|| 1.66| 98.34| 2399|| 0.24| 98.35| > 0.00| 0.00| 0.00 > 4| 0.00| 0.00| 0.00| 0.00|| 1.36| 98.64| 2398|| 0.00| 98.65| > 0.00| 0.00| 0.00 > 14| 0.00| 0.00| 0.00| 0.00|| 1.95| 98.05| 2398|| 0.77| 98.06| > 0.00| 0.00| 0.00 > 5| 0.00| 0.00| 0.00| 0.00|| 1.39| 98.61| 2398|| 0.00| 98.64| > 0.00| 0.00| 0.00 > 15| 0.00| 0.00| 0.00| 0.00|| 8.33| 91.67| 2398|| 7.80| 91.68| > 0.00| 0.00| 0.00 > 6| 0.00| 0.00| 0.00| 0.00|| 1.48| 98.52| 2398|| 0.00| 98.54| > 0.00| 0.00| 0.00 > 16| 0.00| 0.00| 0.00| 0.00|| 2.44| 97.56| 2398|| 1.73| 97.57| > 0.00| 0.00| 0.00 > 7| 0.00| 0.00| 0.00| 0.00|| 2.13| 97.87| 2398|| 0.64| 97.88| > 0.00| 0.00| 0.00 > 17| 0.00| 0.00| 0.00| 0.00|| 1.03| 98.97| 2398|| 0.24| 98.93| > 0.00| 0.00| 0.00 > 8| 0.00| 0.00| 0.00| 0.00|| 1.43| 98.57| 2398|| 0.00| 98.61| > 0.00| 0.00| 0.00 > 18| 0.00| 0.00| 0.00| 0.00|| 1.58| 98.42| 2398|| 0.00| 98.45| > 0.00| 0.00| 0.00 > 9| 0.00| 0.00| 0.00| 0.00|| 1.66| 98.34| 2398|| 0.00| 98.35| > 0.00| 0.00| 0.00 > 19| 0.00| 0.00| 0.00| 0.00|| 1.04| 98.96| 2398|| 0.00| 98.93| > 0.00| 0.00| 0.00 > > -bash-4.2$ sudo /opt/dell/srvadmin/bin/omreport chassis biossetup | > egrep -i "c state|turbo" > Dell Controlled Turbo : Disabled > Turbo Boost : Enabled > Energy Efficient Turbo : Disabled > C States : Disabled > Number of Turbo Boost Enabled Cores for Processor 1 : All > > -bash-4.2$ sudo tail > /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq | grep ^2 > 2399976 > 2399890 > 2399976 > 2399976 > 2399976 > 2399804 > 2399976 > 2399976 > 2400062 > 2399976 > 2399976 > 2399890 > 2399976 > 2400062 > 2399976 > 2399976 > 2399804 > 2399890 > 2399976 > 2399890 > This is what I want as well. My CPUs are 2.1Ghz and I want them to stay there. I've seen latencies of ~1ms when I manage to get the CPUs to stay high clocked, but once they clock down my latency crawls back up to ~1.6ms. > We didn't manage to get this level of consistency until we > used /dev/cpu_dma_latency > (see https://www.kernel.org/doc/Documentation/power/pm_qos_interface.txt) via > tuned: > > -bash-4.2$ sudo tuned-adm active > Current active profile: latency-performance > > We've got some Skylake Ubuntu based hypervisors that we can look at to > compare tomorrow... > Awesome! > Cheers, > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
