Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Dietmar Maurer]

> I'm seeing a lot less retransmit since I have disable pstate.
> (but maybe this is because of bug where frequency was stuck low, and I have
> also a big cluster with a lot of vms)

Please can you verify this?

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Alexandre DERUMIER]

>>What is the suggestion - disbale the pstate driver? I think that would 
>>increase 
>>power consumption? 

>>yes, this is increase power consumption. But increase stability & latencies.

Note that disable pstate, don't disable dynamic frequencies.

we still have classic "cpufreq" governors.

with pstate, we have 2 profiles:
-powersave
-performance 

cpufreq have

-powersave
-ondemand
-performance 


What I don't understand
"
https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt

"Since these policies are implemented in the driver, they are not same as the
cpufreq scaling governors implementation, even if they have the same name in
the cpufreq sysfs (scaling_governors). For example the "performance" policy is
similar to cpufreq’s "performance" governor, but "powersave" is completely
different than the cpufreq "powersave" governor. The strategy here is similar
to cpufreq "ondemand", where the requested P-State is related to the system 
load."
"


for me, pstate "performance is working like cpufreq "ondemand".



- Mail original -
De: "aderumier" <aderum...@odiso.com>
À: "dietmar" <diet...@proxmox.com>
Cc: "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Mardi 20 Septembre 2016 11:05:25
Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
governor

>>What is the suggestion - disbale the pstate driver? I think that would 
>>increase 
>>power consumption? 

yes, this is increase power consumption. But increase stability & latencies. 

my old xeons (previous sandybridge) and amd, always are at maximum cpu 
frequency. 

Redhat have a special daemon "tuned", which have some tunables profiles for 
this (latency,bandwith, ...) 


>>Really? You observed retransmits because of different CPU frequency. I think 
>>retransmits need 
>>much larger speed differences, for example caused by high load (as opposed to 
>>idle nodes). 

I'm seeing a lot less retransmit since I have disable pstate. 
(but maybe this is because of bug where frequency was stuck low, and I have 
also a big cluster with a lot of vms) 


----- Mail original - 
De: "dietmar" <diet...@proxmox.com> 
À: "aderumier" <aderum...@odiso.com>, "pve-devel" <pve-devel@pve.proxmox.com> 
Envoyé: Mardi 20 Septembre 2016 09:52:51 
Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
governor 

> I think this is the normal behavior of pstate, as they are a lower limit. 
> 
> But for virtualisation, I think it's really bad to have changing frequency. 
> (clock problem for example). 

What is the suggestion - disbale the pstate driver? I think that would increase 
power consumption? 

> Also for corosync, that mean that a loaded node will have faster frequency, 
> and non-loaded node low frequency. 
> This can give us reload corosync.conf after removing a node., because low 
> frequency take more time to get the corosync message than the fastest node 
> https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/
>  

Really? You observed retransmits because of different CPU frequency. I think 
retransmits need 
much larger speed differences, for example caused by high load (as opposed to 
idle nodes). 

___ 
pve-devel mailing list 
pve-devel@pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Alexandre DERUMIER]

>>What is the suggestion - disbale the pstate driver? I think that would 
>>increase 
>>power consumption? 

yes, this is increase power consumption. But increase stability & latencies.

my old xeons (previous sandybridge) and amd, always are at maximum cpu 
frequency.

Redhat have a special daemon "tuned", which have some tunables profiles for 
this (latency,bandwith, ...)


>>Really? You observed retransmits because of different CPU frequency. I think 
>>retransmits need 
>>much larger speed differences, for example caused by high load (as opposed to 
>>idle nodes). 

I'm seeing a lot less retransmit since I have disable pstate.
(but maybe this is because of bug where frequency was stuck low, and I have 
also a big cluster with a lot of vms)


- Mail original -
De: "dietmar" <diet...@proxmox.com>
À: "aderumier" <aderum...@odiso.com>, "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Mardi 20 Septembre 2016 09:52:51
Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
governor

> I think this is the normal behavior of pstate, as they are a lower limit. 
> 
> But for virtualisation, I think it's really bad to have changing frequency. 
> (clock problem for example). 

What is the suggestion - disbale the pstate driver? I think that would increase 
power consumption? 

> Also for corosync, that mean that a loaded node will have faster frequency, 
> and non-loaded node low frequency. 
> This can give us reload corosync.conf after removing a node., because low 
> frequency take more time to get the corosync message than the fastest node 
> https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/
>  

Really? You observed retransmits because of different CPU frequency. I think 
retransmits need 
much larger speed differences, for example caused by high load (as opposed to 
idle nodes). 

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Dietmar Maurer]

> I think this is the normal behavior of pstate, as they are a lower limit.
> 
> But for virtualisation, I think it's really bad to have changing frequency.
> (clock problem for example).

What is the suggestion - disbale the pstate driver? I think that would increase
power consumption?
 
> Also for corosync, that mean that a loaded node will have faster frequency,
> and non-loaded node low frequency.
> This can give us reload corosync.conf after removing a node., because low
> frequency take more time to get the corosync message than the fastest node
> https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/

Really? You observed retransmits because of different CPU frequency. I think
retransmits need
much larger speed differences, for example caused by high load (as opposed to
idle nodes).

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Alexandre DERUMIER]

Ok,
I have test the patched kernel,

and I still have the same behavior, the frequency still go up and down.
(But it's not stuck anymore, like the bug I have see last week)

I think this is the normal behavior of pstate, as they are a lower limit.

But for virtualisation, I think it's really bad to have changing frequency. 
(clock problem for example).

Also for corosync, that mean that a loaded node will have faster frequency, and 
non-loaded node low frequency.
This can give us retransmit, because low frequency take more time to get the 
corosync message than the fastest node
https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/


result:

without any load:

root@kvm1:/etc/pve# cat /proc/cpuinfo |grep MHz
cpu MHz : 2025.292
cpu MHz : 1520.816
cpu MHz : 3261.175
cpu MHz : 1875.742
cpu MHz : 2923.445
cpu MHz : 1935.078
cpu MHz : 2860.597
cpu MHz : 1671.820
cpu MHz : 1200.039
cpu MHz : 1653.656
cpu MHz : 1602.433
cpu MHz : 1935.320
cpu MHz : 2042.972
cpu MHz : 1359.761
cpu MHz : 3460.253
cpu MHz : 1200.039
cpu MHz : 2163.097
cpu MHz : 1710.328
cpu MHz : 2249.316
cpu MHz : 1199.675
cpu MHz : 2473.945
cpu MHz : 1731.398
cpu MHz : 2541.273
cpu MHz : 1658.863
cpu MHz : 2528.800
cpu MHz : 1680.660
cpu MHz : 1922.847
cpu MHz : 1369.570
cpu MHz : 1940.890
cpu MHz : 1526.507
cpu MHz : 1952.878
cpu MHz : 1452.761
cpu MHz : 1788.675
cpu MHz : 2137.910
cpu MHz : 1942.828
cpu MHz : 1707.664
cpu MHz : 1438.957
cpu MHz : 1642.757
cpu MHz : 1561.382
cpu MHz : 2104.730

running cpu benchmark:

root@kvm1:/etc/pve# cat /proc/cpuinfo |grep MHz
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902
cpu MHz : 3199.902

- Mail original -
De: "aderumier" <aderum...@odiso.com>
À: "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Lundi 19 Septembre 2016 10:08:45
Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
governor

>>And it's being changed based on cpu load, like actual governor is ondemand. 

From what I read, the intel pstate "performance", have a range min_freq / 
max_freq. 

So it seem to be different than cpufreq "performance", which is "max 
performance" 


(min_freq can be change manually through sysfs, but I don't think they are a 
kernel option to fix to max at boot) 



I'm currently compiling the kernel with patches from Stefan, I'll make a report 
this afternoon. 




- Mail original - 
De: "Dmitry Petuhov" <mityapetu...@gmail.com> 
À: "pve-devel" <pve-devel@pve.proxmox.com> 
Envoyé: Lundi 19 Septembre 2016 09:47:21 
Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
governor 

19.09.2016 01:29, Alexandre DERUMIER wrote: 
> Hi, 
> 
> I have add some strange behaviour of some host last week, (cpu performance 
> degrading) 
> 
> 
> and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency. 
> 
> All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode 
> updates , last proxmox kernel) 
> 
> On the 3 hosts, the frequency was stuck to 800mhz instead 3,1ghz. (note that 
> on other host, the frequency was not stable, up and down between 3,09 && 
> 3,2ghz) 
> 
> Cpu governor is correctly set to max performance in bios + linux. 
> 
> 
> It seem to be a problem with intel pstate driver. 
> 
> 
> I have disabled it with intel_pstate=disable in grub, (also can be disable 
> with CONFIG_X86_INTEL_PSTATE=n) 
> and not frequency is super stable at 3,1ghz 
> 
> 
> Has somebody already seen t

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Alexandre DERUMIER]

>>And it's being changed based on cpu load, like actual governor is ondemand. 

From what I read, the intel pstate "performance", have a range min_freq / 
max_freq.

So it seem to be different than cpufreq "performance", which is "max 
performance"


(min_freq can be change manually through sysfs, but I don't think they are a 
kernel option to fix to max at boot)



I'm currently compiling the kernel with patches from Stefan, I'll make a report 
this afternoon.




- Mail original -
De: "Dmitry Petuhov" <mityapetu...@gmail.com>
À: "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Lundi 19 Septembre 2016 09:47:21
Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
governor

19.09.2016 01:29, Alexandre DERUMIER wrote: 
> Hi, 
> 
> I have add some strange behaviour of some host last week, (cpu performance 
> degrading) 
> 
> 
> and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency. 
> 
> All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode 
> updates , last proxmox kernel) 
> 
> On the 3 hosts, the frequency was stuck to 800mhz instead 3,1ghz. (note that 
> on other host, the frequency was not stable, up and down between 3,09 && 
> 3,2ghz) 
> 
> Cpu governor is correctly set to max performance in bios + linux. 
> 
> 
> It seem to be a problem with intel pstate driver. 
> 
> 
> I have disabled it with intel_pstate=disable in grub, (also can be disable 
> with CONFIG_X86_INTEL_PSTATE=n) 
> and not frequency is super stable at 3,1ghz 
> 
> 
> Has somebody already seen this on xeon v3 ? 
Can confirm that on SandyBridge: 
model name : Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz 
root@msv-spb-pve01:/usr/share/perl5/PVE/Storage# cat 
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq 
1199906 

And it's being changed based on cpu load, like actual governor is ondemand. 

___ 
pve-devel mailing list 
pve-devel@pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Dmitry Petuhov]

19.09.2016 01:29, Alexandre DERUMIER wrote:

Hi,

I have add some strange behaviour of some host last week, (cpu performance 
degrading)


and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency.

All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode 
updates , last proxmox kernel)

On the 3 hosts, the frequency was stuck to 800mhz instead 3,1ghz. (note that on other 
host, the frequency was not stable, up and down between 3,09 && 3,2ghz)

Cpu governor is correctly set to max performance in bios + linux.


It seem to be a problem with intel pstate driver.


I have disabled it with intel_pstate=disable  in grub,  (also can be disable 
with CONFIG_X86_INTEL_PSTATE=n)
and not frequency is super stable at 3,1ghz


Has somebody already seen this on xeon v3 ?

Can confirm that on SandyBridge:
model name  : Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
root@msv-spb-pve01:/usr/share/perl5/PVE/Storage# cat 
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq

1199906

And it's being changed based on cpu load, like actual governor is ondemand.

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Stefan Priebe - Profihost AG]

Am 19.09.2016 um 09:01 schrieb Alexandre DERUMIER:
>>> @alexandre: please can you test if that solves your problem?
> 
> I'll try. does I need to apply the whole patches series ?

normally not. Just apply the whole cpufreq series. If it does not
compile please report to me. I'll point you to the missing piece.

Stefan

> 
> 
> 
> - Mail original -
> De: "dietmar" <diet...@proxmox.com>
> À: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>, "pve-devel" 
> <pve-devel@pve.proxmox.com>
> Cc: "aderumier" <aderum...@odiso.com>
> Envoyé: Lundi 19 Septembre 2016 07:36:57
> Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
> governor
> 
>> The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a 
>> lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq 
>> (also a lot of optimizations) patches in 4.4. 
>>
>> I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and 
>> backports. 
>>
>> See here and look for prefix cpufreq 
>> https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21 
> 
> @alexandre: please can you test if that solves your problem? 
> 
> @stefan: Do you also use the brtfs patches from hhoffstaette/... ? 
> 
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Alexandre DERUMIER]

>>@alexandre: please can you test if that solves your problem?

I'll try. does I need to apply the whole patches series ?



- Mail original -
De: "dietmar" <diet...@proxmox.com>
À: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>, "pve-devel" 
<pve-devel@pve.proxmox.com>
Cc: "aderumier" <aderum...@odiso.com>
Envoyé: Lundi 19 Septembre 2016 07:36:57
Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance 
governor

> The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a 
> lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq 
> (also a lot of optimizations) patches in 4.4. 
> 
> I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and 
> backports. 
> 
> See here and look for prefix cpufreq 
> https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21 

@alexandre: please can you test if that solves your problem? 

@stefan: Do you also use the brtfs patches from hhoffstaette/... ? 

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Stefan Priebe - Profihost AG]

Am 19.09.2016 um 07:36 schrieb Dietmar Maurer:
>> The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a
>> lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq
>> (also a lot of optimizations) patches in 4.4.
>>
>> I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and
>> backports.
>>
>> See here and look for prefix cpufreq
>> https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21
> 
> @alexandre: please can you test if that solves your problem?
> 
> @stefan: Do you also use the brtfs patches from hhoffstaette/... ?

Yes.

Stefan

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Dietmar Maurer]

> The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a
> lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq
> (also a lot of optimizations) patches in 4.4.
> 
> I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and
> backports.
> 
> See here and look for prefix cpufreq
> https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21

@alexandre: please can you test if that solves your problem?

@stefan: Do you also use the brtfs patches from hhoffstaette/... ?

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Stefan Priebe - Profihost AG]

The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a 
lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq 
(also a lot of optimizations) patches in 4.4.

I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and 
backports.

See here and look for prefix cpufreq 
https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21

Greets,
Stefan

Excuse my typo sent from my mobile phone.

> Am 19.09.2016 um 06:49 schrieb Dietmar Maurer :
> 
> 
>> and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency.
>> 
>> All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode
>> updates , last proxmox kernel)
> 
> model name: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz
> 
> but I never observer such problems. I get:
> 
> # cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq 
> 340
> 
> ___
> pve-devel mailing list
> pve-devel@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor

[Dietmar Maurer]

> and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency.
> 
> All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode
> updates , last proxmox kernel)

model name  : Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz

but I never observer such problems. I get:

# cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq 
340

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel