Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
> I'm seeing a lot less retransmit since I have disable pstate. > (but maybe this is because of bug where frequency was stuck low, and I have > also a big cluster with a lot of vms) Please can you verify this? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
>>What is the suggestion - disbale the pstate driver? I think that would >>increase >>power consumption? >>yes, this is increase power consumption. But increase stability & latencies. Note that disable pstate, don't disable dynamic frequencies. we still have classic "cpufreq" governors. with pstate, we have 2 profiles: -powersave -performance cpufreq have -powersave -ondemand -performance What I don't understand " https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt "Since these policies are implemented in the driver, they are not same as the cpufreq scaling governors implementation, even if they have the same name in the cpufreq sysfs (scaling_governors). For example the "performance" policy is similar to cpufreq’s "performance" governor, but "powersave" is completely different than the cpufreq "powersave" governor. The strategy here is similar to cpufreq "ondemand", where the requested P-State is related to the system load." " for me, pstate "performance is working like cpufreq "ondemand". - Mail original - De: "aderumier" <aderum...@odiso.com> À: "dietmar" <diet...@proxmox.com> Cc: "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Mardi 20 Septembre 2016 11:05:25 Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor >>What is the suggestion - disbale the pstate driver? I think that would >>increase >>power consumption? yes, this is increase power consumption. But increase stability & latencies. my old xeons (previous sandybridge) and amd, always are at maximum cpu frequency. Redhat have a special daemon "tuned", which have some tunables profiles for this (latency,bandwith, ...) >>Really? You observed retransmits because of different CPU frequency. I think >>retransmits need >>much larger speed differences, for example caused by high load (as opposed to >>idle nodes). I'm seeing a lot less retransmit since I have disable pstate. (but maybe this is because of bug where frequency was stuck low, and I have also a big cluster with a lot of vms) ----- Mail original - De: "dietmar" <diet...@proxmox.com> À: "aderumier" <aderum...@odiso.com>, "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Mardi 20 Septembre 2016 09:52:51 Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor > I think this is the normal behavior of pstate, as they are a lower limit. > > But for virtualisation, I think it's really bad to have changing frequency. > (clock problem for example). What is the suggestion - disbale the pstate driver? I think that would increase power consumption? > Also for corosync, that mean that a loaded node will have faster frequency, > and non-loaded node low frequency. > This can give us reload corosync.conf after removing a node., because low > frequency take more time to get the corosync message than the fastest node > https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/ > Really? You observed retransmits because of different CPU frequency. I think retransmits need much larger speed differences, for example caused by high load (as opposed to idle nodes). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
>>What is the suggestion - disbale the pstate driver? I think that would >>increase >>power consumption? yes, this is increase power consumption. But increase stability & latencies. my old xeons (previous sandybridge) and amd, always are at maximum cpu frequency. Redhat have a special daemon "tuned", which have some tunables profiles for this (latency,bandwith, ...) >>Really? You observed retransmits because of different CPU frequency. I think >>retransmits need >>much larger speed differences, for example caused by high load (as opposed to >>idle nodes). I'm seeing a lot less retransmit since I have disable pstate. (but maybe this is because of bug where frequency was stuck low, and I have also a big cluster with a lot of vms) - Mail original - De: "dietmar" <diet...@proxmox.com> À: "aderumier" <aderum...@odiso.com>, "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Mardi 20 Septembre 2016 09:52:51 Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor > I think this is the normal behavior of pstate, as they are a lower limit. > > But for virtualisation, I think it's really bad to have changing frequency. > (clock problem for example). What is the suggestion - disbale the pstate driver? I think that would increase power consumption? > Also for corosync, that mean that a loaded node will have faster frequency, > and non-loaded node low frequency. > This can give us reload corosync.conf after removing a node., because low > frequency take more time to get the corosync message than the fastest node > https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/ > Really? You observed retransmits because of different CPU frequency. I think retransmits need much larger speed differences, for example caused by high load (as opposed to idle nodes). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
> I think this is the normal behavior of pstate, as they are a lower limit. > > But for virtualisation, I think it's really bad to have changing frequency. > (clock problem for example). What is the suggestion - disbale the pstate driver? I think that would increase power consumption? > Also for corosync, that mean that a loaded node will have faster frequency, > and non-loaded node low frequency. > This can give us reload corosync.conf after removing a node., because low > frequency take more time to get the corosync message than the fastest node > https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/ Really? You observed retransmits because of different CPU frequency. I think retransmits need much larger speed differences, for example caused by high load (as opposed to idle nodes). ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
Ok, I have test the patched kernel, and I still have the same behavior, the frequency still go up and down. (But it's not stuck anymore, like the bug I have see last week) I think this is the normal behavior of pstate, as they are a lower limit. But for virtualisation, I think it's really bad to have changing frequency. (clock problem for example). Also for corosync, that mean that a loaded node will have faster frequency, and non-loaded node low frequency. This can give us retransmit, because low frequency take more time to get the corosync message than the fastest node https://www.hastexo.com/resources/hints-and-kinks/whats-totem-retransmit-list-all-about-corosync/ result: without any load: root@kvm1:/etc/pve# cat /proc/cpuinfo |grep MHz cpu MHz : 2025.292 cpu MHz : 1520.816 cpu MHz : 3261.175 cpu MHz : 1875.742 cpu MHz : 2923.445 cpu MHz : 1935.078 cpu MHz : 2860.597 cpu MHz : 1671.820 cpu MHz : 1200.039 cpu MHz : 1653.656 cpu MHz : 1602.433 cpu MHz : 1935.320 cpu MHz : 2042.972 cpu MHz : 1359.761 cpu MHz : 3460.253 cpu MHz : 1200.039 cpu MHz : 2163.097 cpu MHz : 1710.328 cpu MHz : 2249.316 cpu MHz : 1199.675 cpu MHz : 2473.945 cpu MHz : 1731.398 cpu MHz : 2541.273 cpu MHz : 1658.863 cpu MHz : 2528.800 cpu MHz : 1680.660 cpu MHz : 1922.847 cpu MHz : 1369.570 cpu MHz : 1940.890 cpu MHz : 1526.507 cpu MHz : 1952.878 cpu MHz : 1452.761 cpu MHz : 1788.675 cpu MHz : 2137.910 cpu MHz : 1942.828 cpu MHz : 1707.664 cpu MHz : 1438.957 cpu MHz : 1642.757 cpu MHz : 1561.382 cpu MHz : 2104.730 running cpu benchmark: root@kvm1:/etc/pve# cat /proc/cpuinfo |grep MHz cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 cpu MHz : 3199.902 - Mail original - De: "aderumier" <aderum...@odiso.com> À: "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Lundi 19 Septembre 2016 10:08:45 Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor >>And it's being changed based on cpu load, like actual governor is ondemand. From what I read, the intel pstate "performance", have a range min_freq / max_freq. So it seem to be different than cpufreq "performance", which is "max performance" (min_freq can be change manually through sysfs, but I don't think they are a kernel option to fix to max at boot) I'm currently compiling the kernel with patches from Stefan, I'll make a report this afternoon. - Mail original - De: "Dmitry Petuhov" <mityapetu...@gmail.com> À: "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Lundi 19 Septembre 2016 09:47:21 Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor 19.09.2016 01:29, Alexandre DERUMIER wrote: > Hi, > > I have add some strange behaviour of some host last week, (cpu performance > degrading) > > > and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency. > > All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode > updates , last proxmox kernel) > > On the 3 hosts, the frequency was stuck to 800mhz instead 3,1ghz. (note that > on other host, the frequency was not stable, up and down between 3,09 && > 3,2ghz) > > Cpu governor is correctly set to max performance in bios + linux. > > > It seem to be a problem with intel pstate driver. > > > I have disabled it with intel_pstate=disable in grub, (also can be disable > with CONFIG_X86_INTEL_PSTATE=n) > and not frequency is super stable at 3,1ghz > > > Has somebody already seen t
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
>>And it's being changed based on cpu load, like actual governor is ondemand. From what I read, the intel pstate "performance", have a range min_freq / max_freq. So it seem to be different than cpufreq "performance", which is "max performance" (min_freq can be change manually through sysfs, but I don't think they are a kernel option to fix to max at boot) I'm currently compiling the kernel with patches from Stefan, I'll make a report this afternoon. - Mail original - De: "Dmitry Petuhov" <mityapetu...@gmail.com> À: "pve-devel" <pve-devel@pve.proxmox.com> Envoyé: Lundi 19 Septembre 2016 09:47:21 Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor 19.09.2016 01:29, Alexandre DERUMIER wrote: > Hi, > > I have add some strange behaviour of some host last week, (cpu performance > degrading) > > > and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency. > > All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode > updates , last proxmox kernel) > > On the 3 hosts, the frequency was stuck to 800mhz instead 3,1ghz. (note that > on other host, the frequency was not stable, up and down between 3,09 && > 3,2ghz) > > Cpu governor is correctly set to max performance in bios + linux. > > > It seem to be a problem with intel pstate driver. > > > I have disabled it with intel_pstate=disable in grub, (also can be disable > with CONFIG_X86_INTEL_PSTATE=n) > and not frequency is super stable at 3,1ghz > > > Has somebody already seen this on xeon v3 ? Can confirm that on SandyBridge: model name : Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz root@msv-spb-pve01:/usr/share/perl5/PVE/Storage# cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq 1199906 And it's being changed based on cpu load, like actual governor is ondemand. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
19.09.2016 01:29, Alexandre DERUMIER wrote: Hi, I have add some strange behaviour of some host last week, (cpu performance degrading) and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency. All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode updates , last proxmox kernel) On the 3 hosts, the frequency was stuck to 800mhz instead 3,1ghz. (note that on other host, the frequency was not stable, up and down between 3,09 && 3,2ghz) Cpu governor is correctly set to max performance in bios + linux. It seem to be a problem with intel pstate driver. I have disabled it with intel_pstate=disable in grub, (also can be disable with CONFIG_X86_INTEL_PSTATE=n) and not frequency is super stable at 3,1ghz Has somebody already seen this on xeon v3 ? Can confirm that on SandyBridge: model name : Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz root@msv-spb-pve01:/usr/share/perl5/PVE/Storage# cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq 1199906 And it's being changed based on cpu load, like actual governor is ondemand. ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
Am 19.09.2016 um 09:01 schrieb Alexandre DERUMIER: >>> @alexandre: please can you test if that solves your problem? > > I'll try. does I need to apply the whole patches series ? normally not. Just apply the whole cpufreq series. If it does not compile please report to me. I'll point you to the missing piece. Stefan > > > > - Mail original - > De: "dietmar" <diet...@proxmox.com> > À: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>, "pve-devel" > <pve-devel@pve.proxmox.com> > Cc: "aderumier" <aderum...@odiso.com> > Envoyé: Lundi 19 Septembre 2016 07:36:57 > Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance > governor > >> The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a >> lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq >> (also a lot of optimizations) patches in 4.4. >> >> I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and >> backports. >> >> See here and look for prefix cpufreq >> https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21 > > @alexandre: please can you test if that solves your problem? > > @stefan: Do you also use the brtfs patches from hhoffstaette/... ? > ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
>>@alexandre: please can you test if that solves your problem? I'll try. does I need to apply the whole patches series ? - Mail original - De: "dietmar" <diet...@proxmox.com> À: "Stefan Priebe, Profihost AG" <s.pri...@profihost.ag>, "pve-devel" <pve-devel@pve.proxmox.com> Cc: "aderumier" <aderum...@odiso.com> Envoyé: Lundi 19 Septembre 2016 07:36:57 Objet: Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor > The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a > lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq > (also a lot of optimizations) patches in 4.4. > > I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and > backports. > > See here and look for prefix cpufreq > https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21 @alexandre: please can you test if that solves your problem? @stefan: Do you also use the brtfs patches from hhoffstaette/... ? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
Am 19.09.2016 um 07:36 schrieb Dietmar Maurer: >> The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a >> lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq >> (also a lot of optimizations) patches in 4.4. >> >> I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and >> backports. >> >> See here and look for prefix cpufreq >> https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21 > > @alexandre: please can you test if that solves your problem? > > @stefan: Do you also use the brtfs patches from hhoffstaette/... ? Yes. Stefan ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
> The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a > lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq > (also a lot of optimizations) patches in 4.4. > > I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and > backports. > > See here and look for prefix cpufreq > https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21 @alexandre: please can you test if that solves your problem? @stefan: Do you also use the brtfs patches from hhoffstaette/... ? ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
The cpufreq and intel pstate driver were somewhat broken in 4.4 there were a lot of changes in 4.5 or 4.6 (can't remember). I'm using around 20 cpufreq (also a lot of optimizations) patches in 4.4. I grabbed those from mr hoffstaette who has his own repo of 4.4 patches and backports. See here and look for prefix cpufreq https://github.com/hhoffstaette/kernel-patches/tree/master/4.4.21 Greets, Stefan Excuse my typo sent from my mobile phone. > Am 19.09.2016 um 06:49 schrieb Dietmar Maurer: > > >> and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency. >> >> All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode >> updates , last proxmox kernel) > > model name: Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz > > but I never observer such problems. I get: > > # cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq > 340 > > ___ > pve-devel mailing list > pve-devel@pve.proxmox.com > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] intel pstate: wrong cpu frequency with performance governor
> and I have found than 3 hosts of my 15 host cluster have wrong cpu frequency. > > All nodes are dell r630, with xeon v3 3,1ghz. (all with last bios/microcode > updates , last proxmox kernel) model name : Intel(R) Xeon(R) CPU E3-1231 v3 @ 3.40GHz but I never observer such problems. I get: # cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq 340 ___ pve-devel mailing list pve-devel@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel