On Mon, Sep 15, 2025 at 08:24:06PM +0300, Ilpo Järvinen wrote:
On Mon, 15 Sep 2025, Lucas De Marchi wrote:
On Mon, Sep 15, 2025 at 12:13:47PM +0300, Ilpo Järvinen wrote:
> pci.c has been used as catch everything that doesn't fits elsewhere
> within PCI core and thus resizable BAR code has been placed there as
> well. Move Resizable BAR related code to a newly introduced rebar.c to
> reduce size of pci.c. After move, there are no pci_rebar_*() calls from
> pci.c indicating this is indeed well-defined subset of PCI core.
>
> Endpoint drivers perform Resizable BAR related operations which could
> well be performed by PCI core to simplify driver-side code. This
> series adds a few new API functions to that effect and converts the
> drivers to use the new APIs (in separate patches).
>
> While at it, also convert BAR sizes bitmask to u64 as PCIe spec already
> specifies more sizes than what will fit u32 to make the API typing more
> future-proof. The extra sizes beyond 128TB are not added at this point.
>
> These are based on pci/main plus a simple "adapter" patch to add the
> include for xe_vram_types.h that was added by a commit in drm-tip.
> Hopefully that is enough to avoid the within context conflict with
> BAR_SIZE_SHIFT removal to let the xe CI tests to be run for this
> series.
>
> There are two minor conflicts with the work in pci/resource but I'm
> hesitant to base this on top of it as this is otherwise entirely
> independent (and would likely prevent GPU CI tests as well). If we end
> up having to pull the bridge window select changes, there should be no
> reason why this does have to become collateral damage (so my
> suggestion, if this is good to go in this cycle, to take this into a
> separate branch than pci/resource and deal with those small conflicts
> while merging into pci/next).
>
> I've tested sysfs resize, i915, and xe BAR resizing functionality. In
> the case of xe, I did small hack patch as its resize is anyway broken
> as is because BAR0 pins the bridge window so resizing BAR2 fails. My
> hack caused other problems further down the road (likely because BAR0
> is in use by the driver so releasing it messed assumptions xe driver
> has) but the BAR resize itself was working which was all I was
is the hack you mention here to release all BARs before attempting the
resize?
Yes, the patch added release of BAR0 prior to resize. The existing xe code
in _resize_bar() already releases BAR2.
During resize, if the first loop in pbus_reassign_bridge_resources()
(called from pci_resize_resource()) finds the bridge window closest to the
endpoint still has a child, it results in having empty saved list because
all upstream bridge windows will then have a child as well.
Empty saved list is checked after the loop and
pbus_reassign_bridge_resources() returns -ENOENT without even trying to
assign the resources. The error is returned even if the actual bridge
window size is large enough to fit the resized resource.
The logic in pci_resize_resource() and pbus_reassign_bridge_resources()
need some other improvements besides that problem, but I likely won't
have time to look at that until completing the fitting algorithm changes.
I'd actually want to add pci_release_and_resize_resource() which would
take care of releasing all the resources of the device (obviously driver
must have its hands off all those BARs when it calls that function). With
the current pci_resize_resource() API, handling the restore of BARs in
case of failure is not as robust as I'd like to make it.
> interested to know. I'm not planning to pursue fixing the pinning
> problem within xe driver because the core changes to consider maximum
> size of the resizable BARs should take care of the main problem by
> different means.
I'd actually like to pursue that myself as that could be propagated to
stable since we do have some resize errors in xe with BMG that I wasn't
understanding. It's likely due to xe_mmio_probe_early() taking a hold of
BAR0 and not expecting it to be moved. We could either remap if we have
have to resize or just move the resize logic early on.
Great. If you have any questions when it comes to the PCI core side code,
please let me know.
I moved the resize to happen before anything else in xe. However when
testing I noticed a scenario failing without involving the driver.
With and without this series I still have the same pass/failure
scenarios:
Tests executed with a BMG. Just after boot, BAR2 is 16GB.
1) If I resize it via sysfs to 8GB and then load the driver, it resizes
it back. Resize from sysfs works too. No change in behavior.
2) If I do "remove the bridge via sysfs and rescan the bus"[1], it fails to
resize (either automatically, on rescan, via sysfs, or loading the xe
driver). It just stays at 256M. The only thing that brings it back is a
reboot. /proc/iomem shows this:
4000000000-7fffffffff : PCI Bus 0000:00
4000000000-44007fffff : PCI Bus 0000:01
4000000000-4017ffffff : PCI Bus 0000:02
4000000000-400fffffff : PCI Bus 0000:03 <<<< BMG
4000000000-400fffffff : 0000:03:00.0
4010000000-40100fffff : PCI Bus 0000:04
4018000000-40187fffff : 0000:01:00.0
And dmesg shows this for the rescan:
[ 1673.189737] pci 0000:01:00.0: [8086:e2ff] type 01 class 0x060400 PCIe Switch
Upstream Port
[ 1673.189794] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x007fffff 64bit pref]
[ 1673.189808] pci 0000:01:00.0: PCI bridge to [bus 00]
[ 1673.189824] pci 0000:01:00.0: bridge window [io 0x0000-0x0fff]
[ 1673.189834] pci 0000:01:00.0: bridge window [mem 0x00000000-0x000fffff]
[ 1673.189856] pci 0000:01:00.0: bridge window [mem 0x00000000-0x000fffff
64bit pref]
[ 1673.189878] pci 0000:01:00.0: Max Payload Size set to 256 (was 128, max 256)
[ 1673.190164] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[ 1673.193531] pci 0000:01:00.0: Adding to iommu group 16
[ 1673.196997] pcieport 0000:00:01.0: ASPM: current common clock configuration
is inconsistent, reconfiguring
[ 1673.197061] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]),
reconfiguring
[ 1673.197421] pci 0000:02:01.0: [8086:e2f0] type 01 class 0x060400 PCIe Switch
Downstream Port
[ 1673.197452] pci 0000:02:01.0: PCI bridge to [bus 00]
[ 1673.197463] pci 0000:02:01.0: bridge window [io 0x0000-0x0fff]
[ 1673.197468] pci 0000:02:01.0: bridge window [mem 0x00000000-0x000fffff]
[ 1673.197482] pci 0000:02:01.0: bridge window [mem 0x00000000-0x000fffff
64bit pref]
[ 1673.197497] pci 0000:02:01.0: Max Payload Size set to 256 (was 128, max 256)
[ 1673.197503] pci 0000:02:01.0: enabling Extended Tags
[ 1673.197660] pci 0000:02:01.0: PME# supported from D0 D3hot D3cold
[ 1673.198411] pci 0000:02:01.0: Adding to iommu group 17
[ 1673.200258] pci 0000:02:02.0: [8086:e2f1] type 01 class 0x060400 PCIe Switch
Downstream Port
[ 1673.200289] pci 0000:02:02.0: PCI bridge to [bus 00]
[ 1673.200299] pci 0000:02:02.0: bridge window [io 0x0000-0x0fff]
[ 1673.200304] pci 0000:02:02.0: bridge window [mem 0x00000000-0x000fffff]
[ 1673.200317] pci 0000:02:02.0: bridge window [mem 0x00000000-0x000fffff
64bit pref]
[ 1673.200333] pci 0000:02:02.0: Max Payload Size set to 256 (was 128, max 256)
[ 1673.200337] pci 0000:02:02.0: enabling Extended Tags
[ 1673.200470] pci 0000:02:02.0: PME# supported from D0 D3hot D3cold
[ 1673.201059] pci 0000:02:02.0: Adding to iommu group 18
[ 1673.202761] pci 0000:01:00.0: PCI bridge to [bus 02-04]
[ 1673.202774] pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]),
reconfiguring
[ 1673.202782] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]),
reconfiguring
[ 1673.203024] pci 0000:03:00.0: [8086:e221] type 00 class 0x030000 PCIe
Endpoint
[ 1673.203060] pci 0000:03:00.0: BAR 0 [mem 0x00000000-0x00ffffff 64bit]
[ 1673.203064] pci 0000:03:00.0: BAR 2 [mem 0x00000000-0x0fffffff 64bit pref]
[ 1673.203069] pci 0000:03:00.0: ROM [mem 0x00000000-0x001fffff pref]
[ 1673.203077] pci 0000:03:00.0: Max Payload Size set to 256 (was 128, max 256)
[ 1673.203209] pci 0000:03:00.0: PME# supported from D0 D3hot
[ 1673.203770] pci 0000:03:00.0: Adding to iommu group 19
[ 1673.205451] pci 0000:03:00.0: vgaarb: setting as boot VGA device
[ 1673.205454] pci 0000:03:00.0: vgaarb: bridge control possible
[ 1673.205455] pci 0000:03:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none
[ 1673.205534] pci 0000:02:01.0: PCI bridge to [bus 03-04]
[ 1673.205543] pci_bus 0000:03: busn_res: [bus 03-04] end is updated to 03
[ 1673.205787] pci 0000:04:00.0: [8086:e2f7] type 00 class 0x040300 PCIe
Endpoint
[ 1673.205848] pci 0000:04:00.0: BAR 0 [mem 0x00000000-0x00003fff 64bit]
[ 1673.205867] pci 0000:04:00.0: Max Payload Size set to 256 (was 128, max 256)
[ 1673.205872] pci 0000:04:00.0: enabling Extended Tags
[ 1673.206012] pci 0000:04:00.0: PME# supported from D3hot D3cold
[ 1673.206528] pci 0000:04:00.0: Adding to iommu group 20
[ 1673.208271] pci 0000:02:02.0: PCI bridge to [bus 04]
[ 1673.208284] pci_bus 0000:04: busn_res: [bus 04] end is updated to 04
[ 1673.208291] pci_bus 0000:02: busn_res: [bus 02-04] end is updated to 04
[ 1673.232003] pcieport 0000:00:01.0: Assigned bridge window [mem
0x83000000-0x840fffff] to [bus 01-04] cannot fit 0x2000000 required for
0000:02:01.0 bridging to [bus 03]
[ 1673.232009] pci 0000:02:01.0: bridge window [mem 0x00000000-0x000fffff] to
[bus 03] requires relaxed alignment rules
[ 1673.232016] pci 0000:02:01.0: bridge window [mem 0x01000000-0x01ffffff] to
[bus 03] add_size 200000 add_align 1000000
[ 1673.232020] pcieport 0000:00:01.0: Assigned bridge window [mem
0x83000000-0x840fffff] to [bus 01-04] cannot fit 0x1800000 required for
0000:01:00.0 bridging to [bus 02-04]
[ 1673.232025] pci 0000:01:00.0: bridge window [mem 0x00000000-0x000fffff] to
[bus 02-04] requires relaxed alignment rules
[ 1673.232027] pcieport 0000:00:01.0: Assigned bridge window [mem
0x83000000-0x840fffff] to [bus 01-04] cannot fit 0x2000000 required for
0000:01:00.0 bridging to [bus 02-04]
[ 1673.232031] pci 0000:01:00.0: bridge window [mem 0x00000000-0x000fffff] to
[bus 02-04] requires relaxed alignment rules
[ 1673.232036] pci 0000:01:00.0: bridge window [mem 0x01000000-0x020fffff] to
[bus 02-04] add_size 200000 add_align 1000000
[ 1673.232077] pci 0000:01:00.0: bridge window [mem 0x4000000000-0x4017ffffff
64bit pref]: assigned
[ 1673.232080] pci 0000:01:00.0: bridge window [mem size 0x01300000]: can't
assign; no space
[ 1673.232082] pci 0000:01:00.0: bridge window [mem size 0x01300000]: failed to
assign
[ 1673.232090] pci 0000:01:00.0: BAR 0 [mem 0x4018000000-0x40187fffff 64bit
pref]: assigned
[ 1673.232103] pci 0000:01:00.0: bridge window [io 0x8000-0x9fff]: assigned
[ 1673.232129] pci 0000:01:00.0: bridge window [mem 0x83000000-0x840fffff]:
assigned
[ 1673.232131] pci 0000:01:00.0: bridge window [mem 0x83000000-0x840fffff]:
failed to expand by 0x200000
[ 1673.232136] pci 0000:01:00.0: bridge window [mem 0x83000000-0x840fffff]:
failed to add optional 200000
[ 1673.232192] pci 0000:02:01.0: bridge window [mem 0x4000000000-0x400fffffff
64bit pref]: assigned
[ 1673.232196] pci 0000:02:01.0: bridge window [mem 0x83000000-0x83ffffff]:
assigned
[ 1673.232200] pci 0000:02:02.0: bridge window [mem 0x84000000-0x840fffff]:
assigned
[ 1673.232202] pci 0000:02:02.0: bridge window [mem 0x4010000000-0x40100fffff
64bit pref]: assigned
[ 1673.232204] pci 0000:02:01.0: bridge window [io 0x8000-0x8fff]: assigned
[ 1673.232206] pci 0000:02:02.0: bridge window [io 0x9000-0x9fff]: assigned
[ 1673.232241] pci 0000:03:00.0: BAR 2 [mem 0x4000000000-0x400fffffff 64bit
pref]: assigned
[ 1673.232250] pci 0000:03:00.0: BAR 0 [mem 0x83000000-0x83ffffff 64bit]:
assigned
[ 1673.232259] pci 0000:03:00.0: ROM [mem size 0x00200000 pref]: can't assign;
no space
[ 1673.232261] pci 0000:03:00.0: ROM [mem size 0x00200000 pref]: failed to
assign
[ 1673.232272] pci 0000:03:00.0: BAR 2 [mem 0x4000000000-0x400fffffff 64bit
pref]: assigned
[ 1673.232280] pci 0000:03:00.0: BAR 0 [mem 0x83000000-0x83ffffff 64bit]:
assigned
[ 1673.232289] pci 0000:03:00.0: ROM [mem size 0x00200000 pref]: can't assign;
no space
[ 1673.232291] pci 0000:03:00.0: ROM [mem size 0x00200000 pref]: failed to
assign
[ 1673.232302] pci 0000:02:01.0: PCI bridge to [bus 03]
[ 1673.232304] pci 0000:02:01.0: bridge window [io 0x8000-0x8fff]
[ 1673.232309] pci 0000:02:01.0: bridge window [mem 0x83000000-0x83ffffff]
[ 1673.232313] pci 0000:02:01.0: bridge window [mem 0x4000000000-0x400fffffff
64bit pref]
[ 1673.232321] pci 0000:04:00.0: BAR 0 [mem 0x84000000-0x84003fff 64bit]:
assigned
[ 1673.232336] pci 0000:02:02.0: PCI bridge to [bus 04]
[ 1673.232339] pci 0000:02:02.0: bridge window [io 0x9000-0x9fff]
[ 1673.232345] pci 0000:02:02.0: bridge window [mem 0x84000000-0x840fffff]
[ 1673.232349] pci 0000:02:02.0: bridge window [mem 0x4010000000-0x40100fffff
64bit pref]
[ 1673.232356] pci 0000:01:00.0: PCI bridge to [bus 02-04]
[ 1673.232359] pci 0000:01:00.0: bridge window [io 0x8000-0x9fff]
[ 1673.232363] pci 0000:01:00.0: bridge window [mem 0x83000000-0x840fffff]
[ 1673.232366] pci 0000:01:00.0: bridge window [mem 0x4000000000-0x4017ffffff
64bit pref]
[ 1673.232471] pcieport 0000:01:00.0: enabling device (0000 -> 0003)
[ 1673.233508] pcieport 0000:02:01.0: enabling device (0000 -> 0003)
[ 1673.233692] pcieport 0000:02:02.0: enabling device (0000 -> 0003)
# echo 9 > /sys/bus/pci/devices/0000\:03\:00.0/resource2_resize
-bash: echo: write error: No space left on device
[1] # echo 1 > /sys/bus/pci/devices/0000\:01\:00.0/remove
# echo 0 > /sys/bus/pci/drivers_autoprobe
# echo 1 > /sys/bus/pci/rescan
I can share the xe patch so you check if it at least fixes it in your
test scenario.
thanks
Lucas De Marchi
--
i.
> Some parts of this are to be used by the resizable BAR changes into the
> resource fitting/assingment logic but these seem to stand on their own
> so sending these out now to reduce the size of the other patch series.
>
> v2:
> - Kerneldoc:
> - Improve formatting of errno returns
> - Open "ctrl" -> "control"
> - Removed mislead "bit" words (when referring to BAR size)
> - Rewrote pci_rebar_get_possible_sizes() kernel doc to not claim the
> returned bitmask is defined in PCIe spec as the capability bits now
> span across two registers in the spec and are not continuous (we
> don't support the second block of bits yet, but this API is expected
> to return the bits without the hole so it will not be matching with
> the spec layout).
> - Dropped superfluous zero check from pci_rebar_size_supported()
> - Small improvement to changelog of patch 7
>
> Ilpo Järvinen (11):
> PCI: Move Resizable BAR code into rebar.c
> PCI: Cleanup pci_rebar_bytes_to_size() and move into rebar.c
> PCI: Move pci_rebar_size_to_bytes() and export it
> PCI: Improve Resizable BAR functions kernel doc
> PCI: Add pci_rebar_size_supported() helper
> drm/i915/gt: Use pci_rebar_size_supported()
> drm/xe/vram: Use PCI rebar helpers in resize_vram_bar()
> PCI: Add pci_rebar_get_max_size()
> drm/xe/vram: Use pci_rebar_get_max_size()
> drm/amdgpu: Use pci_rebar_get_max_size()
> PCI: Convert BAR sizes bitmasks to u64
>
> Documentation/driver-api/pci/pci.rst | 3 +
> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 +-
> drivers/gpu/drm/i915/gt/intel_region_lmem.c | 10 +-
> drivers/gpu/drm/xe/xe_vram.c | 32 +-
> drivers/pci/Makefile | 2 +-
> drivers/pci/iov.c | 9 +-
> drivers/pci/pci-sysfs.c | 2 +-
> drivers/pci/pci.c | 145 ---------
> drivers/pci/pci.h | 5 +-
> drivers/pci/rebar.c | 314 ++++++++++++++++++++
> drivers/pci/setup-res.c | 78 -----
> include/linux/pci.h | 15 +-
> 12 files changed, 350 insertions(+), 273 deletions(-)
> create mode 100644 drivers/pci/rebar.c
>
>
> base-commit: 8f5ae30d69d7543eee0d70083daf4de8fe15d585
> prerequisite-patch-id: 35bd3cd7a60ff7d887450a7fdde73b055a76ae24
> --
> 2.39.5
>
