Re: [vfio-users] higher power usage with vfio-pci then with amdgpu driver
On Sun, 2 Oct 2022 15:03:04 +0200 Pim wrote: > Hey, > > After noticing this commit: > https://github.com/torvalds/linux/commit/7ab5e10eda02da1d9562ffde562c51055d368e9c > and because of high energy prices here, I did some tests with an energy > monitor to see what the power consumption is for my AMD RX480when its > not used. The RX480 is normally only used to pass through to a vm, but > this vm is mostly switched off so the RX480 is not used. In order to get the full benefit from the above commit, you'll need to have a system that supports D3cold, otherwise you'll only see an incremental improvement of also placing root ports in D3hot in addition to the device from previous D3host support. I can't really offer any advice regarding systems that support this, I assume it's common in laptops but not so much in desktops. As the patch series describes, we can't simply use PCI config space registers to enter D3cold, we need hardware and firmware support that's generally invoked through ACPI in order to remove power to the slot. Certainly do check to make sure that all the devices under the root port and the root port itself are also in D3hot. If any devices under the bridge remain in a running state, then runtime PM cannot enter D3cold. > The result is that when I boot my system with vfio-pci enabled it uses > ~90Watt (RX480 driver = vfio-pci), but when I boot my system with > vfio-pci disabled (RX480 driver = amdgpu) then my system uses ~80Watt. > Both times the RX480 is in D3hot power state. > > Any idea why using the vfio-pci driver results in ~10Watt more power > consumption? Probably because the native driver can power down more parts of the device than we can via the generic PCI interface that gets us to D3hot. vfio-pci does not have code to reach a device specific low power state. If slot level power control was available, the referenced commit should entirely power-off the device. Thanks, Alex ___ vfio-users mailing list vfio-users@redhat.com https://listman.redhat.com/mailman/listinfo/vfio-users
[vfio-users] higher power usage with vfio-pci then with amdgpu driver
Hey, After noticing this commit: https://github.com/torvalds/linux/commit/7ab5e10eda02da1d9562ffde562c51055d368e9c and because of high energy prices here, I did some tests with an energy monitor to see what the power consumption is for my AMD RX480when its not used. The RX480 is normally only used to pass through to a vm, but this vm is mostly switched off so the RX480 is not used. The result is that when I boot my system with vfio-pci enabled it uses ~90Watt (RX480 driver = vfio-pci), but when I boot my system with vfio-pci disabled (RX480 driver = amdgpu) then my system uses ~80Watt. Both times the RX480 is in D3hot power state. Any idea why using the vfio-pci driver results in ~10Watt more power consumption? Another weirdest thing is that when I change the drivers after boot using bind/unbind, my system will consistently use another 10W more, ie when testing as follows: - system boots with vfio-pci enabled // -> 90 Watt - echo :02:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind - echo :02:00.0 > /sys/bus/pci/drivers/amdgpu/bind // -> 100Watt - echo :02:00.0 > /sys/bus/pci/drivers/amdgpu/unbind - echo :02:00.0 > /sys/bus/pci/drivers/vfio-pci/bind // -> still 100 Watt even though RX480 is in D3hot state Cheers, Pim Relevant outputs: # uname -a Linux 5.19.11-051911-generic #202209231341 SMP PREEMPT_DYNAMIC Fri Sep 23 14:11:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # dmesg | grep \:02\:00 [ 0.395249] pci :02:00.0: [1002:67df] type 00 class 0x03 [ 0.395305] pci :02:00.0: reg 0x10: [mem 0xa000-0xafff 64bit pref] [ 0.395362] pci :02:00.0: reg 0x18: [mem 0xb000-0xb01f 64bit pref] [ 0.395412] pci :02:00.0: reg 0x20: [io 0xd000-0xd0ff] [ 0.395453] pci :02:00.0: reg 0x24: [mem 0xfbd0-0xfbd3] [ 0.395496] pci :02:00.0: reg 0x30: [mem 0xfbd4-0xfbd5 pref] [ 0.395536] pci :02:00.0: enabling Extended Tags [ 0.395656] pci :02:00.0: supports D1 D2 [ 0.395689] pci :02:00.0: PME# supported from D1 D2 D3hot D3cold [ 0.395860] pci :02:00.1: [1002:aaf0] type 00 class 0x040300 [ 0.395920] pci :02:00.1: reg 0x10: [mem 0xfbd6-0xfbd63fff 64bit] [ 0.396018] pci :02:00.1: enabling Extended Tags [ 0.396112] pci :02:00.1: supports D1 D2 [ 0.409125] pci :02:00.0: vgaarb: setting as boot VGA device [ 0.409125] pci :02:00.0: vgaarb: bridge control possible [ 0.409125] pci :02:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none [ 0.470935] pci :02:00.0: CLS mismatch (64 != 32), using 64 bytes [ 0.470996] pci :02:00.1: D0 power state depends on :02:00.0 [ 0.475972] pci :02:00.0: Adding to iommu group 46 [ 0.476027] pci :02:00.1: Adding to iommu group 46 [ 1.074885] vfio-pci :02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 9.174548] vfio-pci :02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none # cat /sys/bus/pci/devices/\:02\:00.0/d3cold_allowed 1 # cat /sys/bus/pci/devices/\:02\:00.0/power_state D3hot # cat /sys/bus/pci/devices/\:02\:00.0/uevent DRIVER=vfio-pci PCI_CLASS=3 PCI_ID=1002:67DF PCI_SUBSYS_ID=1462:3413 PCI_SLOT_NAME=:02:00.0 MODALIAS=pci:v1002d67DFsv1462sd3413bc03sc00i00 ___ vfio-users mailing list vfio-users@redhat.com https://listman.redhat.com/mailman/listinfo/vfio-users
