Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-17 Thread Wido den Hollander 


On 05/16/2018 03:34 PM, Wido den Hollander wrote:
> 
> 
> On 05/16/2018 01:22 PM, Blair Bethwaite wrote:
>> On 15 May 2018 at 08:45, Wido den Hollander > > wrote:
>>
>> > We've got some Skylake Ubuntu based hypervisors that we can look at to
>> > compare tomorrow...
>> > 
>>
>> Awesome!
>>
>>
>> Ok, so results still inconclusive I'm afraid...
>>
>> The Ubuntu machines we're looking at (Dell R740s and C6420s running with
>> Performance BIOS power profile, which amongst other things disables
>> cstates and enables turbo) are currently running either a 4.13 or a 4.15
>> HWE kernel - we needed 4.13 to support PERC10 and even get them booting
>> from local storage, then 4.15 to get around a prlimit bug that was
>> breaking Nova snapshots, so here we are. Where are you getting 4.16,
>> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16/ ?
>>
> 
> Yes, that's where I got 4.16: 4.16.7-041607-generic
> 
> I also tried 4.16.8, but that didn't change anything either.
> 
> Server I am testing with are these:
> https://www.supermicro.nl/products/system/1U/1029/SYS-1029U-TN10RT.cfm
> 
>> So interestingly in our case we seem to have no cpufreq driver loaded.
>> After installing linux-generic-tools (cause cpupower is supposed to
>> supersede cpufrequtils I think?):
>>
>> rr42-03:~$ uname -a
>> Linux rcgpudc1rr42-03 4.15.0-13-generic #14~16.04.1-Ubuntu SMP Sat Mar
>> 17 03:04:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>
>> rr42-03:~$ cat /proc/cmdline
>> BOOT_IMAGE=/vmlinuz-4.15.0-13-generic root=/dev/mapper/vg00-root ro
>> intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=1
>>
> 
> I have those settings as well, intel_idle and processor.max_cstate.
> 
> [1.776036] intel_idle: disabled
> 

Ok, so this caught my attention, why was that module disabled? So I
started to dig deeper and deeper.

It seems that max_cstate=0 for intel_idle actually seems to disable it.

So I'm now using: 'processor.max_cstate=1 intel_idle.max_cstate=1'

I ran my tests again and here is what I get with fio and iodepth=1 with
a randwrite test of 4k. Below is a list of kernels and the usec latency:

- 4.4: 1957
- 4.13: 1259
- 4.16: 918

Again, this is with a Dual Xeon Silver 4110 with NVMe SSDs.

With the performance governor the CPUs now stay pinned on 2.4Ghz:

analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 0.97 ms.
  hardware limits: 800 MHz - 3.00 GHz
  available cpufreq governors: performance, powersave
  current policy: frequency should be within 800 MHz and 3.00 GHz.
  The governor "performance" may decide which speed to use
  within this range.
  current CPU frequency is 2.40 GHz.

Going to test further with Ceph OSD options and such, but this seems
like a lot better result.

> That works, the CPUs stay in C0 or C1 according to i7z, but they are
> clocking down in Mhz, for example:
> 
> processor : 23
> vendor_id : GenuineIntel
> cpu family: 6
> model : 85
> model name: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
> stepping  : 4
> microcode : 0x243
> cpu MHz   : 799.953
> 
> /sys/devices/system/cpu/intel_pstate/min_perf_pct is set to 100, but
> that setting doesn't seem to do anything.
> 
> I'm running out of ideas :-)
> 
> Wido
> 
>> rr42-03:~$ lscpu
>> Architecture:          x86_64
>> CPU op-mode(s):        32-bit, 64-bit
>> Byte Order:            Little Endian
>> CPU(s):                36
>> On-line CPU(s) list:   0-35
>> Thread(s) per core:    1
>> Core(s) per socket:    18
>> Socket(s):             2
>> NUMA node(s):          2
>> Vendor ID:             GenuineIntel
>> CPU family:            6
>> Model:                 85
>> Model name:            Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
>> Stepping:              4
>> CPU MHz:               3400.956
>> BogoMIPS:              5401.45
>> Virtualization:        VT-x
>> L1d cache:             32K
>> L1i cache:             32K
>> L2 cache:              1024K
>> L3 cache:              25344K
>> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34
>> NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35
>> Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
>> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts
>> rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq
>> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
>> dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx
>> f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3
>> invpcid_single pti intel_ppin mba tpr_shadow vnmi flexpriority ept vpid
>> fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx
>> rdt_a avx512f avx512dq rds

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-16 Thread Alexandre DERUMIER 
Hi,

I'm able to have fixed frequency with

intel_pstate=disable intel_idle.max_cstate=0 processor.max_cstate=1 

Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz

# cat /proc/cpuinfo |grep MHz
cpu MHz : 3400.002
cpu MHz : 3399.994
cpu MHz : 3399.995
cpu MHz : 3399.994
cpu MHz : 3399.997
cpu MHz : 3399.998
cpu MHz : 3399.992
cpu MHz : 3399.989
cpu MHz : 3399.998
cpu MHz : 3399.994
cpu MHz : 3399.988
cpu MHz : 3399.987
cpu MHz : 3399.990
cpu MHz : 3399.990
cpu MHz : 3399.994
cpu MHz : 3399.996
cpu MHz : 3399.996
cpu MHz : 3399.985
cpu MHz : 3399.991
cpu MHz : 3399.981
cpu MHz : 3399.979
cpu MHz : 3399.993
cpu MHz : 3399.985
cpu MHz : 3399.985


- Mail original -
De: "Wido den Hollander" 
À: "Blair Bethwaite" 
Cc: "ceph-users" , "Nick Fisk" 
Envoyé: Mercredi 16 Mai 2018 15:34:35
Objet: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on 
NVMe/SSD Ceph OSDs

On 05/16/2018 01:22 PM, Blair Bethwaite wrote: 
> On 15 May 2018 at 08:45, Wido den Hollander  <mailto:w...@42on.com>> wrote: 
> 
> > We've got some Skylake Ubuntu based hypervisors that we can look at to 
> > compare tomorrow... 
> > 
> 
> Awesome! 
> 
> 
> Ok, so results still inconclusive I'm afraid... 
> 
> The Ubuntu machines we're looking at (Dell R740s and C6420s running with 
> Performance BIOS power profile, which amongst other things disables 
> cstates and enables turbo) are currently running either a 4.13 or a 4.15 
> HWE kernel - we needed 4.13 to support PERC10 and even get them booting 
> from local storage, then 4.15 to get around a prlimit bug that was 
> breaking Nova snapshots, so here we are. Where are you getting 4.16, 
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16/ ? 
> 

Yes, that's where I got 4.16: 4.16.7-041607-generic 

I also tried 4.16.8, but that didn't change anything either. 

Server I am testing with are these: 
https://www.supermicro.nl/products/system/1U/1029/SYS-1029U-TN10RT.cfm 

> So interestingly in our case we seem to have no cpufreq driver loaded. 
> After installing linux-generic-tools (cause cpupower is supposed to 
> supersede cpufrequtils I think?): 
> 
> rr42-03:~$ uname -a 
> Linux rcgpudc1rr42-03 4.15.0-13-generic #14~16.04.1-Ubuntu SMP Sat Mar 
> 17 03:04:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux 
> 
> rr42-03:~$ cat /proc/cmdline 
> BOOT_IMAGE=/vmlinuz-4.15.0-13-generic root=/dev/mapper/vg00-root ro 
> intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=1 
> 

I have those settings as well, intel_idle and processor.max_cstate. 

[ 1.776036] intel_idle: disabled 

That works, the CPUs stay in C0 or C1 according to i7z, but they are 
clocking down in Mhz, for example: 

processor : 23 
vendor_id : GenuineIntel 
cpu family : 6 
model : 85 
model name : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz 
stepping : 4 
microcode : 0x243 
cpu MHz : 799.953 

/sys/devices/system/cpu/intel_pstate/min_perf_pct is set to 100, but 
that setting doesn't seem to do anything. 

I'm running out of ideas :-) 

Wido 

> rr42-03:~$ lscpu 
> Architecture: x86_64 
> CPU op-mode(s): 32-bit, 64-bit 
> Byte Order: Little Endian 
> CPU(s): 36 
> On-line CPU(s) list: 0-35 
> Thread(s) per core: 1 
> Core(s) per socket: 18 
> Socket(s): 2 
> NUMA node(s): 2 
> Vendor ID: GenuineIntel 
> CPU family: 6 
> Model: 85 
> Model name: Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz 
> Stepping: 4 
> CPU MHz: 3400.956 
> BogoMIPS: 5401.45 
> Virtualization: VT-x 
> L1d cache: 32K 
> L1i cache: 32K 
> L2 cache: 1024K 
> L3 cache: 25344K 
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34 
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35 
> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr 
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
> syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts 
> rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq 
> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid 
> dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx 
> f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 
> invpcid_single pti intel_ppin mba tpr_shadow vnmi flexpriority ept vpid 
> fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx 
> rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd 
> avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc 
> cqm_mbm_total cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts pku

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-16 Thread Wido den Hollander 


On 05/16/2018 01:22 PM, Blair Bethwaite wrote:
> On 15 May 2018 at 08:45, Wido den Hollander  > wrote:
> 
> > We've got some Skylake Ubuntu based hypervisors that we can look at to
> > compare tomorrow...
> > 
> 
> Awesome!
> 
> 
> Ok, so results still inconclusive I'm afraid...
> 
> The Ubuntu machines we're looking at (Dell R740s and C6420s running with
> Performance BIOS power profile, which amongst other things disables
> cstates and enables turbo) are currently running either a 4.13 or a 4.15
> HWE kernel - we needed 4.13 to support PERC10 and even get them booting
> from local storage, then 4.15 to get around a prlimit bug that was
> breaking Nova snapshots, so here we are. Where are you getting 4.16,
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16/ ?
> 

Yes, that's where I got 4.16: 4.16.7-041607-generic

I also tried 4.16.8, but that didn't change anything either.

Server I am testing with are these:
https://www.supermicro.nl/products/system/1U/1029/SYS-1029U-TN10RT.cfm

> So interestingly in our case we seem to have no cpufreq driver loaded.
> After installing linux-generic-tools (cause cpupower is supposed to
> supersede cpufrequtils I think?):
> 
> rr42-03:~$ uname -a
> Linux rcgpudc1rr42-03 4.15.0-13-generic #14~16.04.1-Ubuntu SMP Sat Mar
> 17 03:04:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> 
> rr42-03:~$ cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-4.15.0-13-generic root=/dev/mapper/vg00-root ro
> intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=1
> 

I have those settings as well, intel_idle and processor.max_cstate.

[1.776036] intel_idle: disabled

That works, the CPUs stay in C0 or C1 according to i7z, but they are
clocking down in Mhz, for example:

processor   : 23
vendor_id   : GenuineIntel
cpu family  : 6
model   : 85
model name  : Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
stepping: 4
microcode   : 0x243
cpu MHz : 799.953

/sys/devices/system/cpu/intel_pstate/min_perf_pct is set to 100, but
that setting doesn't seem to do anything.

I'm running out of ideas :-)

Wido

> rr42-03:~$ lscpu
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                36
> On-line CPU(s) list:   0-35
> Thread(s) per core:    1
> Core(s) per socket:    18
> Socket(s):             2
> NUMA node(s):          2
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 85
> Model name:            Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
> Stepping:              4
> CPU MHz:               3400.956
> BogoMIPS:              5401.45
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              1024K
> L3 cache:              25344K
> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34
> NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35
> Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts
> rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq
> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
> dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx
> f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3
> invpcid_single pti intel_ppin mba tpr_shadow vnmi flexpriority ept vpid
> fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx
> rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd
> avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
> cqm_mbm_total cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts pku
> ospke
> 
> rr42-03:~$ sudo cpupower frequency-info
> analyzing CPU 0:
>   no or unknown cpufreq driver is active on this CPU
>   CPUs which run at the same hardware frequency: Not Available
>   CPUs which need to have their frequency coordinated by software: Not
> Available
>   maximum transition latency:  Cannot determine or is not supported.
> Not Available
>   available cpufreq governors: Not Available
>   Unable to determine current policy
>   current CPU frequency: Unable to call hardware
>   current CPU frequency:  Unable to call to kernel
>   boost state support:
>     Supported: yes
>     Active: yes
> 
> 
> And of course there is nothing under sysfs (/sys/devices/system/cpu*).
> But /proc/cpuinfo and cpupower-monitor show that we seem to be hitting
> turbo freqs:
> 
> rr42-03:~$ sudo cpupower monitor
>               |Nehalem                    || Mperf
> PKG |CORE|CPU | C3   | C6   | PC3  | PC6  || C0   | Cx   | Freq
>    0|   0|   0|  0.00|  0.00|  0.00|  0.00||  0.05| 99.95|  3391
>    0|   1|   4|  0.00|  0.00|  0.00|  0.00||  0.02| 99.98|  3389
>    0|   2|   8|  0.00|  0.00|  0.00|  0.00||  0.14| 99.86| 

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-16 Thread Blair Bethwaite 
Possibly, but I think they'd be using the V100s rather than the CPUs.

For reference:

rr42-03:~$ sudo cpupower monitor -l
Monitor "Nehalem" (4 states) - Might overflow after 92200 s
C3  [C] -> Processor Core C3
C6  [C] -> Processor Core C6
PC3 [P] -> Processor Package C3
PC6 [P] -> Processor Package C6
Monitor "Mperf" (3 states) - Might overflow after 92200 s
C0  [T] -> Processor Core not idle
Cx  [T] -> Processor Core in an idle state
Freq[T] -> Average Frequency (including boost) in MHz

So that node is doing nothing much right now.


On 16 May 2018 at 04:29, John Hearns  wrote:

> Blair,
>methinks someone is doing bitcoin mining on your systems when they are
> idle   :-)
>
> I WAS going to say that maybe the cpupower utility needs an update to cope
> with that generation of CPUs.
> But 7proc/cpuinfo never lies  (does it ?)
>
>
>
>
> On 16 May 2018 at 13:22, Blair Bethwaite 
> wrote:
>
>> On 15 May 2018 at 08:45, Wido den Hollander  wrote:
>>>
>>> > We've got some Skylake Ubuntu based hypervisors that we can look at to
>>> > compare tomorrow...
>>> >
>>>
>>> Awesome!
>>
>>
>> Ok, so results still inconclusive I'm afraid...
>>
>> The Ubuntu machines we're looking at (Dell R740s and C6420s running with
>> Performance BIOS power profile, which amongst other things disables cstates
>> and enables turbo) are currently running either a 4.13 or a 4.15 HWE kernel
>> - we needed 4.13 to support PERC10 and even get them booting from local
>> storage, then 4.15 to get around a prlimit bug that was breaking Nova
>> snapshots, so here we are. Where are you getting 4.16,
>> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16/ ?
>>
>> So interestingly in our case we seem to have no cpufreq driver loaded.
>> After installing linux-generic-tools (cause cpupower is supposed to
>> supersede cpufrequtils I think?):
>>
>> rr42-03:~$ uname -a
>> Linux rcgpudc1rr42-03 4.15.0-13-generic #14~16.04.1-Ubuntu SMP Sat Mar 17
>> 03:04:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>>
>> rr42-03:~$ cat /proc/cmdline
>> BOOT_IMAGE=/vmlinuz-4.15.0-13-generic root=/dev/mapper/vg00-root ro
>> intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=1
>>
>> rr42-03:~$ lscpu
>> Architecture:  x86_64
>> CPU op-mode(s):32-bit, 64-bit
>> Byte Order:Little Endian
>> CPU(s):36
>> On-line CPU(s) list:   0-35
>> Thread(s) per core:1
>> Core(s) per socket:18
>> Socket(s): 2
>> NUMA node(s):  2
>> Vendor ID: GenuineIntel
>> CPU family:6
>> Model: 85
>> Model name:Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
>> Stepping:  4
>> CPU MHz:   3400.956
>> BogoMIPS:  5401.45
>> Virtualization:VT-x
>> L1d cache: 32K
>> L1i cache: 32K
>> L2 cache:  1024K
>> L3 cache:  25344K
>> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34
>> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35
>> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
>> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts
>> rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64
>> monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca
>> sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c
>> rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3
>> invpcid_single pti intel_ppin mba tpr_shadow vnmi flexpriority ept vpid
>> fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a
>> avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw
>> avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
>> cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts pku ospke
>>
>> rr42-03:~$ sudo cpupower frequency-info
>> analyzing CPU 0:
>>   no or unknown cpufreq driver is active on this CPU
>>   CPUs which run at the same hardware frequency: Not Available
>>   CPUs which need to have their frequency coordinated by software: Not
>> Available
>>   maximum transition latency:  Cannot determine or is not supported.
>> Not Available
>>   available cpufreq governors: Not Available
>>   Unable to determine current policy
>>   current CPU frequency: Unable to call hardware
>>   current CPU frequency:  Unable to call to kernel
>>   boost state support:
>> Supported: yes
>> Active: yes
>>
>>
>> And of course there is nothing under sysfs (/sys/devices/system/cpu*).
>> But /proc/cpuinfo and cpupower-monitor show that we seem to be hitting
>> turbo freqs:
>>
>> rr42-03:~$ sudo cpupower monitor
>>   |Nehalem|| Mperf
>> PKG |CORE|CPU | C3   | C6   | PC3  | PC6  || C0   | Cx   | Freq
>>0|   0|   0|  0.00|  0.00|  0.00|  0.00||  0.05| 99.95|  3391
>>

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-16 Thread John Hearns 
Blair,
   methinks someone is doing bitcoin mining on your systems when they are
idle   :-)

I WAS going to say that maybe the cpupower utility needs an update to cope
with that generation of CPUs.
But 7proc/cpuinfo never lies  (does it ?)




On 16 May 2018 at 13:22, Blair Bethwaite  wrote:

> On 15 May 2018 at 08:45, Wido den Hollander  wrote:
>>
>> > We've got some Skylake Ubuntu based hypervisors that we can look at to
>> > compare tomorrow...
>> >
>>
>> Awesome!
>
>
> Ok, so results still inconclusive I'm afraid...
>
> The Ubuntu machines we're looking at (Dell R740s and C6420s running with
> Performance BIOS power profile, which amongst other things disables cstates
> and enables turbo) are currently running either a 4.13 or a 4.15 HWE kernel
> - we needed 4.13 to support PERC10 and even get them booting from local
> storage, then 4.15 to get around a prlimit bug that was breaking Nova
> snapshots, so here we are. Where are you getting 4.16,
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16/ ?
>
> So interestingly in our case we seem to have no cpufreq driver loaded.
> After installing linux-generic-tools (cause cpupower is supposed to
> supersede cpufrequtils I think?):
>
> rr42-03:~$ uname -a
> Linux rcgpudc1rr42-03 4.15.0-13-generic #14~16.04.1-Ubuntu SMP Sat Mar 17
> 03:04:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> rr42-03:~$ cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-4.15.0-13-generic root=/dev/mapper/vg00-root ro
> intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=1
>
> rr42-03:~$ lscpu
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):36
> On-line CPU(s) list:   0-35
> Thread(s) per core:1
> Core(s) per socket:18
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 85
> Model name:Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
> Stepping:  4
> CPU MHz:   3400.956
> BogoMIPS:  5401.45
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  1024K
> L3 cache:  25344K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35
> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts
> rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64
> monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca
> sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c
> rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3
> invpcid_single pti intel_ppin mba tpr_shadow vnmi flexpriority ept vpid
> fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a
> avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw
> avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
> cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts pku ospke
>
> rr42-03:~$ sudo cpupower frequency-info
> analyzing CPU 0:
>   no or unknown cpufreq driver is active on this CPU
>   CPUs which run at the same hardware frequency: Not Available
>   CPUs which need to have their frequency coordinated by software: Not
> Available
>   maximum transition latency:  Cannot determine or is not supported.
> Not Available
>   available cpufreq governors: Not Available
>   Unable to determine current policy
>   current CPU frequency: Unable to call hardware
>   current CPU frequency:  Unable to call to kernel
>   boost state support:
> Supported: yes
> Active: yes
>
>
> And of course there is nothing under sysfs (/sys/devices/system/cpu*). But
> /proc/cpuinfo and cpupower-monitor show that we seem to be hitting turbo
> freqs:
>
> rr42-03:~$ sudo cpupower monitor
>   |Nehalem|| Mperf
> PKG |CORE|CPU | C3   | C6   | PC3  | PC6  || C0   | Cx   | Freq
>0|   0|   0|  0.00|  0.00|  0.00|  0.00||  0.05| 99.95|  3391
>0|   1|   4|  0.00|  0.00|  0.00|  0.00||  0.02| 99.98|  3389
>0|   2|   8|  0.00|  0.00|  0.00|  0.00||  0.14| 99.86|  3067
>0|   3|   6|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3385
>0|   4|   2|  0.00|  0.00|  0.00|  0.00||  0.09| 99.91|  3119
>0|   8|  12|  0.00|  0.00|  0.00|  0.00||  0.03| 99.97|  3312
>0|   9|  16|  0.00|  0.00|  0.00|  0.00||  0.11| 99.89|  3157
>0|  10|  14|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3352
>0|  11|  10|  0.00|  0.00|  0.00|  0.00||  0.05| 99.95|  3390
>0|  16|  20|  0.00|  0.00|  0.00|  0.00||  0.00|100.00|  3387
>0|  17|  24|  0.00|  0.00|  0.00|  0.00||  0.22| 99.78|  3115
>0|  18|  26|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3389
>0|  1

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-16 Thread Blair Bethwaite 
On 15 May 2018 at 08:45, Wido den Hollander  wrote:
>
> > We've got some Skylake Ubuntu based hypervisors that we can look at to
> > compare tomorrow...
> >
>
> Awesome!


Ok, so results still inconclusive I'm afraid...

The Ubuntu machines we're looking at (Dell R740s and C6420s running with
Performance BIOS power profile, which amongst other things disables cstates
and enables turbo) are currently running either a 4.13 or a 4.15 HWE kernel
- we needed 4.13 to support PERC10 and even get them booting from local
storage, then 4.15 to get around a prlimit bug that was breaking Nova
snapshots, so here we are. Where are you getting 4.16,
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16/ ?

So interestingly in our case we seem to have no cpufreq driver loaded.
After installing linux-generic-tools (cause cpupower is supposed to
supersede cpufrequtils I think?):

rr42-03:~$ uname -a
Linux rcgpudc1rr42-03 4.15.0-13-generic #14~16.04.1-Ubuntu SMP Sat Mar 17
03:04:59 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

rr42-03:~$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.15.0-13-generic root=/dev/mapper/vg00-root ro
intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=1

rr42-03:~$ lscpu
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):36
On-line CPU(s) list:   0-35
Thread(s) per core:1
Core(s) per socket:18
Socket(s): 2
NUMA node(s):  2
Vendor ID: GenuineIntel
CPU family:6
Model: 85
Model name:Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
Stepping:  4
CPU MHz:   3400.956
BogoMIPS:  5401.45
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  1024K
L3 cache:  25344K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall
nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl
vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic
movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm
3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin
mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2
smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap
clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1
xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local ibpb ibrs stibp
dtherm ida arat pln pts pku ospke

rr42-03:~$ sudo cpupower frequency-info
analyzing CPU 0:
  no or unknown cpufreq driver is active on this CPU
  CPUs which run at the same hardware frequency: Not Available
  CPUs which need to have their frequency coordinated by software: Not
Available
  maximum transition latency:  Cannot determine or is not supported.
Not Available
  available cpufreq governors: Not Available
  Unable to determine current policy
  current CPU frequency: Unable to call hardware
  current CPU frequency:  Unable to call to kernel
  boost state support:
Supported: yes
Active: yes


And of course there is nothing under sysfs (/sys/devices/system/cpu*). But
/proc/cpuinfo and cpupower-monitor show that we seem to be hitting turbo
freqs:

rr42-03:~$ sudo cpupower monitor
  |Nehalem|| Mperf
PKG |CORE|CPU | C3   | C6   | PC3  | PC6  || C0   | Cx   | Freq
   0|   0|   0|  0.00|  0.00|  0.00|  0.00||  0.05| 99.95|  3391
   0|   1|   4|  0.00|  0.00|  0.00|  0.00||  0.02| 99.98|  3389
   0|   2|   8|  0.00|  0.00|  0.00|  0.00||  0.14| 99.86|  3067
   0|   3|   6|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3385
   0|   4|   2|  0.00|  0.00|  0.00|  0.00||  0.09| 99.91|  3119
   0|   8|  12|  0.00|  0.00|  0.00|  0.00||  0.03| 99.97|  3312
   0|   9|  16|  0.00|  0.00|  0.00|  0.00||  0.11| 99.89|  3157
   0|  10|  14|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3352
   0|  11|  10|  0.00|  0.00|  0.00|  0.00||  0.05| 99.95|  3390
   0|  16|  20|  0.00|  0.00|  0.00|  0.00||  0.00|100.00|  3387
   0|  17|  24|  0.00|  0.00|  0.00|  0.00||  0.22| 99.78|  3115
   0|  18|  26|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3389
   0|  19|  22|  0.00|  0.00|  0.00|  0.00||  0.00|100.00|  3366
   0|  20|  18|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3392
   0|  24|  28|  0.00|  0.00|  0.00|  0.00||  0.00|100.00|  3376
   0|  25|  32|  0.00|  0.00|  0.00|  0.00||  0.05| 99.95|  3390
   0|  26|  34|  0.00|  0.00|  0.00|  0.00||  0.03| 99.97|  3391
   0|  27|  30|  0.00|  0.00|  0.00|  0.00||  0.01| 99.99|  3392
   1|   0|   1|  0.00|  0.00|  0.00|  0.00||  0.00|100.00|  3394
   1|   1|   5|  0.00|  0.00|  0.00|  0.

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-15 Thread Wido den Hollander 


On 05/15/2018 02:51 PM, Blair Bethwaite wrote:
> Sorry, bit late to get back to this...
> 
> On Wed., 2 May 2018, 06:19 Nick Fisk,  > wrote:
> 
> 4.16 required?
> 
> 
> Looks like it - thanks for pointing that out.
> 
> Wido, I don't think you are doing anything wrong here, maybe this is a
> bug...
> 
> I've got RHEL7 + Broadwell based Ceph nodes here for which the same
> tuning appears to be working fine:
> 

Odd indeed. Keep in mind that I indeed have the newer Intel Scalable
CPUs with Ubuntu 16.04 and a 4.16 kernel.

My main goal is the lowest possible latency with NVMe and for that you
need higher clock speeds.

(more down)

> -bash-4.2$ lsb_release -a
> LSB Version:    :core-4.1-amd64:core-4.1-noarch
> Distributor ID: RedHatEnterpriseServer
> Description:    Red Hat Enterprise Linux Server release 7.3 (Maipo)
> Release:        7.3
> Codename:       Maipo
> 
> -bash-4.2$ lscpu
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                20
> On-line CPU(s) list:   0-19
> Thread(s) per core:    2
> Core(s) per socket:    10
> Socket(s):             1
> NUMA node(s):          1
> Vendor ID:             GenuineIntel
> CPU family:            6
> Model:                 79
> Model name:            Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:              1
> CPU MHz:               2745.960
> BogoMIPS:              4399.83
> Virtualization:        VT-x
> L1d cache:             32K
> L1i cache:             32K
> L2 cache:              256K
> L3 cache:              25600K
> NUMA node0 CPU(s):     0-19
> 
> -bash-4.2$ cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-3.10.0-514.2.2.el7.x86_64
> root=/dev/mapper/vg00-LogVol00 ro nofb splash=quiet crashkernel=auto
> rd.lvm.lv =vg00/LogVol00 rd.lvm.lv
> =vg00/LogVol01 rhgb quiet LANG=en_US.UTF-8
> 
> -bash-4.2$ sudo cpupower frequency-info
> analyzing CPU 0:
>   driver: intel_pstate
>   CPUs which run at the same hardware frequency: 0
>   CPUs which need to have their frequency coordinated by software: 0
>   maximum transition latency:  Cannot determine or is not supported.
>   hardware limits: 1.20 GHz - 3.10 GHz
>   available cpufreq governors: performance powersave
>   current policy: frequency should be within 1.20 GHz and 3.10 GHz.
>                   The governor "performance" may decide which speed to use
>                   within this range.
>   current CPU frequency: 2.40 GHz (asserted by call to hardware)
>   boost state support:
>     Supported: yes
>     Active: yes
> 
> -bash-4.2$ sudo cpupower -c 0-19 monitor
>     |Nehalem                    || Mperf              || Idle_Stats
> CPU | C3   | C6   | PC3  | PC6  || C0   | Cx   | Freq || POLL | C1-B |
> C1E- | C3-B | C6-B
>    0|  0.00|  0.00|  0.00|  0.00|| 20.93| 79.07|  2398||  1.00| 79.08| 
> 0.00|  0.00|  0.00
>   10|  0.00|  0.00|  0.00|  0.00||  1.81| 98.19|  2398||  0.00| 98.23| 
> 0.00|  0.00|  0.00
>    1|  0.00|  0.00|  0.00|  0.00||  3.80| 96.20|  2398||  2.10| 96.21| 
> 0.00|  0.00|  0.00
>   11|  0.00|  0.00|  0.00|  0.00||  7.95| 92.05|  2398||  7.59| 92.06| 
> 0.00|  0.00|  0.00
>    2|  0.00|  0.00|  0.00|  0.00||  1.99| 98.01|  2398||  0.00| 98.04| 
> 0.00|  0.00|  0.00
>   12|  0.00|  0.00|  0.00|  0.00||  1.59| 98.41|  2398||  0.64| 98.42| 
> 0.00|  0.00|  0.00
>    3|  0.00|  0.00|  0.00|  0.00|| 24.58| 75.42|  2398||  0.00| 75.43| 
> 0.00|  0.00|  0.00
>   13|  0.00|  0.00|  0.00|  0.00||  1.66| 98.34|  2399||  0.24| 98.35| 
> 0.00|  0.00|  0.00
>    4|  0.00|  0.00|  0.00|  0.00||  1.36| 98.64|  2398||  0.00| 98.65| 
> 0.00|  0.00|  0.00
>   14|  0.00|  0.00|  0.00|  0.00||  1.95| 98.05|  2398||  0.77| 98.06| 
> 0.00|  0.00|  0.00
>    5|  0.00|  0.00|  0.00|  0.00||  1.39| 98.61|  2398||  0.00| 98.64| 
> 0.00|  0.00|  0.00
>   15|  0.00|  0.00|  0.00|  0.00||  8.33| 91.67|  2398||  7.80| 91.68| 
> 0.00|  0.00|  0.00
>    6|  0.00|  0.00|  0.00|  0.00||  1.48| 98.52|  2398||  0.00| 98.54| 
> 0.00|  0.00|  0.00
>   16|  0.00|  0.00|  0.00|  0.00||  2.44| 97.56|  2398||  1.73| 97.57| 
> 0.00|  0.00|  0.00
>    7|  0.00|  0.00|  0.00|  0.00||  2.13| 97.87|  2398||  0.64| 97.88| 
> 0.00|  0.00|  0.00
>   17|  0.00|  0.00|  0.00|  0.00||  1.03| 98.97|  2398||  0.24| 98.93| 
> 0.00|  0.00|  0.00
>    8|  0.00|  0.00|  0.00|  0.00||  1.43| 98.57|  2398||  0.00| 98.61| 
> 0.00|  0.00|  0.00
>   18|  0.00|  0.00|  0.00|  0.00||  1.58| 98.42|  2398||  0.00| 98.45| 
> 0.00|  0.00|  0.00
>    9|  0.00|  0.00|  0.00|  0.00||  1.66| 98.34|  2398||  0.00| 98.35| 
> 0.00|  0.00|  0.00
>   19|  0.00|  0.00|  0.00|  0.00||  1.04| 98.96|  2398||  0.00| 98.93| 
> 0.00|  0.00|  0.00
> 
> -bash-4.2$ sudo /opt/dell/srvadmin/bin/omreport chassis biossetup |
> egrep -i "c state|turbo"
> Dell Controlled Turbo                               : Disabled
> Turbo Boost                                         : Enabled
> Energy Efficient Turbo                              :

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-15 Thread Blair Bethwaite 
Sorry, bit late to get back to this...

On Wed., 2 May 2018, 06:19 Nick Fisk,  wrote:

> 4.16 required?
>

Looks like it - thanks for pointing that out.

Wido, I don't think you are doing anything wrong here, maybe this is a
bug...

I've got RHEL7 + Broadwell based Ceph nodes here for which the same tuning
appears to be working fine:

-bash-4.2$ lsb_release -a
LSB Version::core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:Red Hat Enterprise Linux Server release 7.3 (Maipo)
Release:7.3
Codename:   Maipo

-bash-4.2$ lscpu
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):20
On-line CPU(s) list:   0-19
Thread(s) per core:2
Core(s) per socket:10
Socket(s): 1
NUMA node(s):  1
Vendor ID: GenuineIntel
CPU family:6
Model: 79
Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Stepping:  1
CPU MHz:   2745.960
BogoMIPS:  4399.83
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  25600K
NUMA node0 CPU(s): 0-19

-bash-4.2$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.2.2.el7.x86_64
root=/dev/mapper/vg00-LogVol00 ro nofb splash=quiet crashkernel=auto
rd.lvm.lv=vg00/LogVol00 rd.lvm.lv=vg00/LogVol01 rhgb quiet LANG=en_US.UTF-8

-bash-4.2$ sudo cpupower frequency-info
analyzing CPU 0:
  driver: intel_pstate
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.20 GHz - 3.10 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 1.20 GHz and 3.10 GHz.
  The governor "performance" may decide which speed to use
  within this range.
  current CPU frequency: 2.40 GHz (asserted by call to hardware)
  boost state support:
Supported: yes
Active: yes

-bash-4.2$ sudo cpupower -c 0-19 monitor
|Nehalem|| Mperf  || Idle_Stats
CPU | C3   | C6   | PC3  | PC6  || C0   | Cx   | Freq || POLL | C1-B | C1E-
| C3-B | C6-B
   0|  0.00|  0.00|  0.00|  0.00|| 20.93| 79.07|  2398||  1.00| 79.08|
0.00|  0.00|  0.00
  10|  0.00|  0.00|  0.00|  0.00||  1.81| 98.19|  2398||  0.00| 98.23|
0.00|  0.00|  0.00
   1|  0.00|  0.00|  0.00|  0.00||  3.80| 96.20|  2398||  2.10| 96.21|
0.00|  0.00|  0.00
  11|  0.00|  0.00|  0.00|  0.00||  7.95| 92.05|  2398||  7.59| 92.06|
0.00|  0.00|  0.00
   2|  0.00|  0.00|  0.00|  0.00||  1.99| 98.01|  2398||  0.00| 98.04|
0.00|  0.00|  0.00
  12|  0.00|  0.00|  0.00|  0.00||  1.59| 98.41|  2398||  0.64| 98.42|
0.00|  0.00|  0.00
   3|  0.00|  0.00|  0.00|  0.00|| 24.58| 75.42|  2398||  0.00| 75.43|
0.00|  0.00|  0.00
  13|  0.00|  0.00|  0.00|  0.00||  1.66| 98.34|  2399||  0.24| 98.35|
0.00|  0.00|  0.00
   4|  0.00|  0.00|  0.00|  0.00||  1.36| 98.64|  2398||  0.00| 98.65|
0.00|  0.00|  0.00
  14|  0.00|  0.00|  0.00|  0.00||  1.95| 98.05|  2398||  0.77| 98.06|
0.00|  0.00|  0.00
   5|  0.00|  0.00|  0.00|  0.00||  1.39| 98.61|  2398||  0.00| 98.64|
0.00|  0.00|  0.00
  15|  0.00|  0.00|  0.00|  0.00||  8.33| 91.67|  2398||  7.80| 91.68|
0.00|  0.00|  0.00
   6|  0.00|  0.00|  0.00|  0.00||  1.48| 98.52|  2398||  0.00| 98.54|
0.00|  0.00|  0.00
  16|  0.00|  0.00|  0.00|  0.00||  2.44| 97.56|  2398||  1.73| 97.57|
0.00|  0.00|  0.00
   7|  0.00|  0.00|  0.00|  0.00||  2.13| 97.87|  2398||  0.64| 97.88|
0.00|  0.00|  0.00
  17|  0.00|  0.00|  0.00|  0.00||  1.03| 98.97|  2398||  0.24| 98.93|
0.00|  0.00|  0.00
   8|  0.00|  0.00|  0.00|  0.00||  1.43| 98.57|  2398||  0.00| 98.61|
0.00|  0.00|  0.00
  18|  0.00|  0.00|  0.00|  0.00||  1.58| 98.42|  2398||  0.00| 98.45|
0.00|  0.00|  0.00
   9|  0.00|  0.00|  0.00|  0.00||  1.66| 98.34|  2398||  0.00| 98.35|
0.00|  0.00|  0.00
  19|  0.00|  0.00|  0.00|  0.00||  1.04| 98.96|  2398||  0.00| 98.93|
0.00|  0.00|  0.00

-bash-4.2$ sudo /opt/dell/srvadmin/bin/omreport chassis biossetup | egrep
-i "c state|turbo"
Dell Controlled Turbo   : Disabled
Turbo Boost : Enabled
Energy Efficient Turbo  : Disabled
C States: Disabled
Number of Turbo Boost Enabled Cores for Processor 1 : All

-bash-4.2$ sudo tail /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
| grep ^2
2399976
2399890
2399976
2399976
2399976
2399804
2399976
2399976
2400062
2399976
2399976
2399890
2399976
2400062
2399976
2399976
2399804
2399890
2399976
2399890

We didn't manage to get this level of consistency until we
used /dev/cpu_dma_latency (see
https://www.kernel.org/doc/Documentation/power/pm_qos_interface.txt) via
tuned:

-bash-4.2$ sudo tuned-adm active
Current activ

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-15 Thread Wido den Hollander 


On 05/14/2018 04:46 PM, Nick Fisk wrote:
> Hi Wido,
> 
> Are you trying this setting?
> 
> /sys/devices/system/cpu/intel_pstate/min_perf_pct
> 

Yes, but that doesn't help. I can set it to 80, 100 or any value I like,
the CPUs keep clocking down to 800Mhz.

At first I was having some issues with getting intel_pstate loaded, but
with 4.16 it loaded without any problems, but still, CPUs keep clocking
down.

Wido

> 
> 
> -Original Message-
> From: ceph-users  On Behalf Of Wido den
> Hollander
> Sent: 14 May 2018 14:14
> To: n...@fisk.me.uk; 'Blair Bethwaite' 
> Cc: 'ceph-users' 
> Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on
> NVMe/SSD Ceph OSDs
> 
> 
> 
> On 05/01/2018 10:19 PM, Nick Fisk wrote:
>> 4.16 required?
>> https://www.phoronix.com/scan.php?page=news_item&px=Skylake-X-P-State-
>> Linux-
>> 4.16
>>
> 
> I've been trying with the 4.16 kernel for the last few days, but still, it's
> not working.
> 
> The CPU's keep clocking down to 800Mhz
> 
> I've set scaling_min_freq=scaling_max_freq in /sys, but that doesn't change
> a thing. The CPUs keep scaling down.
> 
> Still not close to the 1ms latency with these CPUs :(
> 
> Wido
> 
>>
>> -----Original Message-----
>> From: ceph-users  On Behalf Of 
>> Blair Bethwaite
>> Sent: 01 May 2018 16:46
>> To: Wido den Hollander 
>> Cc: ceph-users ; Nick Fisk 
>> 
>> Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency 
>> scaling on NVMe/SSD Ceph OSDs
>>
>> Also curious about this over here. We've got a rack's worth of R740XDs 
>> with Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active 
>> on them, though I don't believe they are any different at the OS level 
>> to our Broadwell nodes (where it is loaded).
>>
>> Have you tried poking the kernel's pmqos interface for your use-case?
>>
>> On 2 May 2018 at 01:07, Wido den Hollander  wrote:
>>> Hi,
>>>
>>> I've been trying to get the lowest latency possible out of the new 
>>> Xeon Scalable CPUs and so far I got down to 1.3ms with the help of Nick.
>>>
>>> However, I can't seem to pin the CPUs to always run at their maximum 
>>> frequency.
>>>
>>> If I disable power saving in the BIOS they stay at 2.1Ghz (Silver 
>>> 4110), but that disables the boost.
>>>
>>> With the Power Saving enabled in the BIOS and when giving the OS all 
>>> control for some reason the CPUs keep scaling down.
>>>
>>> $ echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>>>
>>> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report 
>>> errors and bugs to cpuf...@vger.kernel.org, please.
>>> analyzing CPU 0:
>>>   driver: intel_pstate
>>>   CPUs which run at the same hardware frequency: 0
>>>   CPUs which need to have their frequency coordinated by software: 0
>>>   maximum transition latency: 0.97 ms.
>>>   hardware limits: 800 MHz - 3.00 GHz
>>>   available cpufreq governors: performance, powersave
>>>   current policy: frequency should be within 800 MHz and 3.00 GHz.
>>>   The governor "performance" may decide which speed to
> use
>>>   within this range.
>>>   current CPU frequency is 800 MHz.
>>>
>>> I do see the CPUs scale up to 2.1Ghz, but they quickly scale down 
>>> again to 800Mhz and that hurts latency. (50% difference!)
>>>
>>> With the CPUs scaling down to 800Mhz my latency jumps from 1.3ms to 
>>> 2.4ms on avg. With turbo enabled I hope to get down to 1.1~1.2ms on avg.
>>>
>>> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
>>> performance
>>>
>>> Everything seems to be OK and I would expect the CPUs to stay at 
>>> 2.10Ghz, but they aren't.
>>>
>>> C-States are also pinned to 0 as a boot parameter for the kernel:
>>>
>>> processor.max_cstate=1 intel_idle.max_cstate=0
>>>
>>> Running Ubuntu 16.04.4 with the 4.13 kernel from the HWE from Ubuntu.
>>>
>>> Has anybody tried this yet with the recent Intel Xeon Scalable CPUs?
>>>
>>> Thanks,
>>>
>>> Wido
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> Cheers,
>> ~Blairo
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-14 Thread Nick Fisk 
Hi Wido,

Are you trying this setting?

/sys/devices/system/cpu/intel_pstate/min_perf_pct



-Original Message-
From: ceph-users  On Behalf Of Wido den
Hollander
Sent: 14 May 2018 14:14
To: n...@fisk.me.uk; 'Blair Bethwaite' 
Cc: 'ceph-users' 
Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on
NVMe/SSD Ceph OSDs



On 05/01/2018 10:19 PM, Nick Fisk wrote:
> 4.16 required?
> https://www.phoronix.com/scan.php?page=news_item&px=Skylake-X-P-State-
> Linux-
> 4.16
> 

I've been trying with the 4.16 kernel for the last few days, but still, it's
not working.

The CPU's keep clocking down to 800Mhz

I've set scaling_min_freq=scaling_max_freq in /sys, but that doesn't change
a thing. The CPUs keep scaling down.

Still not close to the 1ms latency with these CPUs :(

Wido

> 
> -Original Message-
> From: ceph-users  On Behalf Of 
> Blair Bethwaite
> Sent: 01 May 2018 16:46
> To: Wido den Hollander 
> Cc: ceph-users ; Nick Fisk 
> 
> Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency 
> scaling on NVMe/SSD Ceph OSDs
> 
> Also curious about this over here. We've got a rack's worth of R740XDs 
> with Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active 
> on them, though I don't believe they are any different at the OS level 
> to our Broadwell nodes (where it is loaded).
> 
> Have you tried poking the kernel's pmqos interface for your use-case?
> 
> On 2 May 2018 at 01:07, Wido den Hollander  wrote:
>> Hi,
>>
>> I've been trying to get the lowest latency possible out of the new 
>> Xeon Scalable CPUs and so far I got down to 1.3ms with the help of Nick.
>>
>> However, I can't seem to pin the CPUs to always run at their maximum 
>> frequency.
>>
>> If I disable power saving in the BIOS they stay at 2.1Ghz (Silver 
>> 4110), but that disables the boost.
>>
>> With the Power Saving enabled in the BIOS and when giving the OS all 
>> control for some reason the CPUs keep scaling down.
>>
>> $ echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>>
>> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report 
>> errors and bugs to cpuf...@vger.kernel.org, please.
>> analyzing CPU 0:
>>   driver: intel_pstate
>>   CPUs which run at the same hardware frequency: 0
>>   CPUs which need to have their frequency coordinated by software: 0
>>   maximum transition latency: 0.97 ms.
>>   hardware limits: 800 MHz - 3.00 GHz
>>   available cpufreq governors: performance, powersave
>>   current policy: frequency should be within 800 MHz and 3.00 GHz.
>>   The governor "performance" may decide which speed to
use
>>   within this range.
>>   current CPU frequency is 800 MHz.
>>
>> I do see the CPUs scale up to 2.1Ghz, but they quickly scale down 
>> again to 800Mhz and that hurts latency. (50% difference!)
>>
>> With the CPUs scaling down to 800Mhz my latency jumps from 1.3ms to 
>> 2.4ms on avg. With turbo enabled I hope to get down to 1.1~1.2ms on avg.
>>
>> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
>> performance
>>
>> Everything seems to be OK and I would expect the CPUs to stay at 
>> 2.10Ghz, but they aren't.
>>
>> C-States are also pinned to 0 as a boot parameter for the kernel:
>>
>> processor.max_cstate=1 intel_idle.max_cstate=0
>>
>> Running Ubuntu 16.04.4 with the 4.13 kernel from the HWE from Ubuntu.
>>
>> Has anybody tried this yet with the recent Intel Xeon Scalable CPUs?
>>
>> Thanks,
>>
>> Wido
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> --
> Cheers,
> ~Blairo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-14 Thread John Hearns 
Wido, I am going to put my rather large foot in it here.
I am sure it is understood that the Turbo mode will not keep all cores at
the maximum frequency at any given time.
There is a thermal envelope for the chip, and the chip works to keep  the
power dissipation within that envelope.
>From what I gather there is a range of thermal limits even within a given
processor SKU, so every chip will exhibit
different Turbo mode behaviour.
And I am sure we all know that when AVX comes into use the Turbo limit is
lower.

I guess what I am saying that for to have reproducible behaviour, if you
care about it for timings etc. Turbo
can be switched off.
Before you say it, in this case you want to achieve the minimum latency and
reproducibility at the Mhz level is not important.

Also worth saying that cooling is important with Turboboost comes into
play. I heard a paper at an HPC Advisory Council
where a Russian setup by Lenovo got significantly more performance at the
HPC acceptance testing stage when cooling was turned up.

I guess my rambling has not added much to this debate, sorry.
cue a friendly Intel engineer to wander in and tell us exactly what is
going on.



On 14 May 2018 at 15:13, Wido den Hollander  wrote:

>
>
> On 05/01/2018 10:19 PM, Nick Fisk wrote:
> > 4.16 required?
> > https://www.phoronix.com/scan.php?page=news_item&px=Skylake-
> X-P-State-Linux-
> > 4.16
> >
>
> I've been trying with the 4.16 kernel for the last few days, but still,
> it's not working.
>
> The CPU's keep clocking down to 800Mhz
>
> I've set scaling_min_freq=scaling_max_freq in /sys, but that doesn't
> change a thing. The CPUs keep scaling down.
>
> Still not close to the 1ms latency with these CPUs :(
>
> Wido
>
> >
> > -Original Message-
> > From: ceph-users  On Behalf Of Blair
> > Bethwaite
> > Sent: 01 May 2018 16:46
> > To: Wido den Hollander 
> > Cc: ceph-users ; Nick Fisk 
> > Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling
> on
> > NVMe/SSD Ceph OSDs
> >
> > Also curious about this over here. We've got a rack's worth of R740XDs
> with
> > Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active on them,
> > though I don't believe they are any different at the OS level to our
> > Broadwell nodes (where it is loaded).
> >
> > Have you tried poking the kernel's pmqos interface for your use-case?
> >
> > On 2 May 2018 at 01:07, Wido den Hollander  wrote:
> >> Hi,
> >>
> >> I've been trying to get the lowest latency possible out of the new
> >> Xeon Scalable CPUs and so far I got down to 1.3ms with the help of Nick.
> >>
> >> However, I can't seem to pin the CPUs to always run at their maximum
> >> frequency.
> >>
> >> If I disable power saving in the BIOS they stay at 2.1Ghz (Silver
> >> 4110), but that disables the boost.
> >>
> >> With the Power Saving enabled in the BIOS and when giving the OS all
> >> control for some reason the CPUs keep scaling down.
> >>
> >> $ echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
> >>
> >> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report
> >> errors and bugs to cpuf...@vger.kernel.org, please.
> >> analyzing CPU 0:
> >>   driver: intel_pstate
> >>   CPUs which run at the same hardware frequency: 0
> >>   CPUs which need to have their frequency coordinated by software: 0
> >>   maximum transition latency: 0.97 ms.
> >>   hardware limits: 800 MHz - 3.00 GHz
> >>   available cpufreq governors: performance, powersave
> >>   current policy: frequency should be within 800 MHz and 3.00 GHz.
> >>   The governor "performance" may decide which speed to
> use
> >>   within this range.
> >>   current CPU frequency is 800 MHz.
> >>
> >> I do see the CPUs scale up to 2.1Ghz, but they quickly scale down
> >> again to 800Mhz and that hurts latency. (50% difference!)
> >>
> >> With the CPUs scaling down to 800Mhz my latency jumps from 1.3ms to
> >> 2.4ms on avg. With turbo enabled I hope to get down to 1.1~1.2ms on avg.
> >>
> >> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> >> performance
> >>
> >> Everything seems to be OK and I would expect the CPUs to stay at
> >> 2.10Ghz, but they aren't.
> >>
> >> C-States are also pinned to 0 as a boot parameter for the kernel:
> >>
> >> processor.max_cstate=1 intel_

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-14 Thread Wido den Hollander 


On 05/01/2018 10:19 PM, Nick Fisk wrote:
> 4.16 required?
> https://www.phoronix.com/scan.php?page=news_item&px=Skylake-X-P-State-Linux-
> 4.16
> 

I've been trying with the 4.16 kernel for the last few days, but still,
it's not working.

The CPU's keep clocking down to 800Mhz

I've set scaling_min_freq=scaling_max_freq in /sys, but that doesn't
change a thing. The CPUs keep scaling down.

Still not close to the 1ms latency with these CPUs :(

Wido

> 
> -Original Message-
> From: ceph-users  On Behalf Of Blair
> Bethwaite
> Sent: 01 May 2018 16:46
> To: Wido den Hollander 
> Cc: ceph-users ; Nick Fisk 
> Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on
> NVMe/SSD Ceph OSDs
> 
> Also curious about this over here. We've got a rack's worth of R740XDs with
> Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active on them,
> though I don't believe they are any different at the OS level to our
> Broadwell nodes (where it is loaded).
> 
> Have you tried poking the kernel's pmqos interface for your use-case?
> 
> On 2 May 2018 at 01:07, Wido den Hollander  wrote:
>> Hi,
>>
>> I've been trying to get the lowest latency possible out of the new 
>> Xeon Scalable CPUs and so far I got down to 1.3ms with the help of Nick.
>>
>> However, I can't seem to pin the CPUs to always run at their maximum 
>> frequency.
>>
>> If I disable power saving in the BIOS they stay at 2.1Ghz (Silver 
>> 4110), but that disables the boost.
>>
>> With the Power Saving enabled in the BIOS and when giving the OS all 
>> control for some reason the CPUs keep scaling down.
>>
>> $ echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>>
>> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report 
>> errors and bugs to cpuf...@vger.kernel.org, please.
>> analyzing CPU 0:
>>   driver: intel_pstate
>>   CPUs which run at the same hardware frequency: 0
>>   CPUs which need to have their frequency coordinated by software: 0
>>   maximum transition latency: 0.97 ms.
>>   hardware limits: 800 MHz - 3.00 GHz
>>   available cpufreq governors: performance, powersave
>>   current policy: frequency should be within 800 MHz and 3.00 GHz.
>>   The governor "performance" may decide which speed to use
>>   within this range.
>>   current CPU frequency is 800 MHz.
>>
>> I do see the CPUs scale up to 2.1Ghz, but they quickly scale down 
>> again to 800Mhz and that hurts latency. (50% difference!)
>>
>> With the CPUs scaling down to 800Mhz my latency jumps from 1.3ms to 
>> 2.4ms on avg. With turbo enabled I hope to get down to 1.1~1.2ms on avg.
>>
>> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
>> performance
>>
>> Everything seems to be OK and I would expect the CPUs to stay at 
>> 2.10Ghz, but they aren't.
>>
>> C-States are also pinned to 0 as a boot parameter for the kernel:
>>
>> processor.max_cstate=1 intel_idle.max_cstate=0
>>
>> Running Ubuntu 16.04.4 with the 4.13 kernel from the HWE from Ubuntu.
>>
>> Has anybody tried this yet with the recent Intel Xeon Scalable CPUs?
>>
>> Thanks,
>>
>> Wido
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> --
> Cheers,
> ~Blairo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-01 Thread Nick Fisk 
4.16 required?
https://www.phoronix.com/scan.php?page=news_item&px=Skylake-X-P-State-Linux-
4.16


-Original Message-
From: ceph-users  On Behalf Of Blair
Bethwaite
Sent: 01 May 2018 16:46
To: Wido den Hollander 
Cc: ceph-users ; Nick Fisk 
Subject: Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on
NVMe/SSD Ceph OSDs

Also curious about this over here. We've got a rack's worth of R740XDs with
Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active on them,
though I don't believe they are any different at the OS level to our
Broadwell nodes (where it is loaded).

Have you tried poking the kernel's pmqos interface for your use-case?

On 2 May 2018 at 01:07, Wido den Hollander  wrote:
> Hi,
>
> I've been trying to get the lowest latency possible out of the new 
> Xeon Scalable CPUs and so far I got down to 1.3ms with the help of Nick.
>
> However, I can't seem to pin the CPUs to always run at their maximum 
> frequency.
>
> If I disable power saving in the BIOS they stay at 2.1Ghz (Silver 
> 4110), but that disables the boost.
>
> With the Power Saving enabled in the BIOS and when giving the OS all 
> control for some reason the CPUs keep scaling down.
>
> $ echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>
> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report 
> errors and bugs to cpuf...@vger.kernel.org, please.
> analyzing CPU 0:
>   driver: intel_pstate
>   CPUs which run at the same hardware frequency: 0
>   CPUs which need to have their frequency coordinated by software: 0
>   maximum transition latency: 0.97 ms.
>   hardware limits: 800 MHz - 3.00 GHz
>   available cpufreq governors: performance, powersave
>   current policy: frequency should be within 800 MHz and 3.00 GHz.
>   The governor "performance" may decide which speed to use
>   within this range.
>   current CPU frequency is 800 MHz.
>
> I do see the CPUs scale up to 2.1Ghz, but they quickly scale down 
> again to 800Mhz and that hurts latency. (50% difference!)
>
> With the CPUs scaling down to 800Mhz my latency jumps from 1.3ms to 
> 2.4ms on avg. With turbo enabled I hope to get down to 1.1~1.2ms on avg.
>
> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> performance
>
> Everything seems to be OK and I would expect the CPUs to stay at 
> 2.10Ghz, but they aren't.
>
> C-States are also pinned to 0 as a boot parameter for the kernel:
>
> processor.max_cstate=1 intel_idle.max_cstate=0
>
> Running Ubuntu 16.04.4 with the 4.13 kernel from the HWE from Ubuntu.
>
> Has anybody tried this yet with the recent Intel Xeon Scalable CPUs?
>
> Thanks,
>
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

2018-05-01 Thread Blair Bethwaite 
Also curious about this over here. We've got a rack's worth of R740XDs
with Xeon 4114's running RHEL 7.4 and intel-pstate isn't even active
on them, though I don't believe they are any different at the OS level
to our Broadwell nodes (where it is loaded).

Have you tried poking the kernel's pmqos interface for your use-case?

On 2 May 2018 at 01:07, Wido den Hollander  wrote:
> Hi,
>
> I've been trying to get the lowest latency possible out of the new Xeon
> Scalable CPUs and so far I got down to 1.3ms with the help of Nick.
>
> However, I can't seem to pin the CPUs to always run at their maximum
> frequency.
>
> If I disable power saving in the BIOS they stay at 2.1Ghz (Silver 4110),
> but that disables the boost.
>
> With the Power Saving enabled in the BIOS and when giving the OS all
> control for some reason the CPUs keep scaling down.
>
> $ echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct
>
> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
> Report errors and bugs to cpuf...@vger.kernel.org, please.
> analyzing CPU 0:
>   driver: intel_pstate
>   CPUs which run at the same hardware frequency: 0
>   CPUs which need to have their frequency coordinated by software: 0
>   maximum transition latency: 0.97 ms.
>   hardware limits: 800 MHz - 3.00 GHz
>   available cpufreq governors: performance, powersave
>   current policy: frequency should be within 800 MHz and 3.00 GHz.
>   The governor "performance" may decide which speed to use
>   within this range.
>   current CPU frequency is 800 MHz.
>
> I do see the CPUs scale up to 2.1Ghz, but they quickly scale down again
> to 800Mhz and that hurts latency. (50% difference!)
>
> With the CPUs scaling down to 800Mhz my latency jumps from 1.3ms to
> 2.4ms on avg. With turbo enabled I hope to get down to 1.1~1.2ms on avg.
>
> $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> performance
>
> Everything seems to be OK and I would expect the CPUs to stay at
> 2.10Ghz, but they aren't.
>
> C-States are also pinned to 0 as a boot parameter for the kernel:
>
> processor.max_cstate=1 intel_idle.max_cstate=0
>
> Running Ubuntu 16.04.4 with the 4.13 kernel from the HWE from Ubuntu.
>
> Has anybody tried this yet with the recent Intel Xeon Scalable CPUs?
>
> Thanks,
>
> Wido
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com