This is a follow-up to:
https://mail-archive.com/[email protected]/msg152555.html
Hi,
I appear to have another instance of this multi-composite-resource
bridge-window undersizing regression, this time with two discrete NVIDIA
GPUs behind AMD chipset PCIe switches.
The allocation arithmetic looks equivalent to the original report, but I
have not yet tested Ilpo's newer patch series. Please treat this as an
additional likely instance rather than confirmation that the proposed
series fixes this machine.
The failure was reproduced on Ubuntu 26.04 with its 7.0.0-31-generic
kernel. The same hardware allocated correctly with Ubuntu's
6.17.0-42-generic kernel.
Summary
=======
On Linux 7.0, forcing bridge resource reassignment gives the shared parent
window only the plain sum of two composite child windows. The second child
cannot start immediately after the first because both child windows inherit
the alignment of their largest BAR.
The later sibling bridge, 0000:0b:08.0, consequently receives neither its
prefetchable nor non-prefetchable window. The NVIDIA GPU at 0000:11:00.0
then has BAR0, BAR1, BAR3, its ROM, and its audio-function BAR unassigned.
The NVIDIA driver subsequently reports:
NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
NVRM: BAR0 is 0M @ 0x0 (PCI:0000:11:00.0)
nvidia 0000:11:00.0: probe with driver nvidia failed with error -1
The BAR allocation failure occurs before the NVIDIA driver probes, so this
does not appear to originate in the NVIDIA driver.
System
======
Distribution:
Ubuntu 26.04.1 LTS (Resolute)
Failing kernel:
linux-image-7.0.0-31-generic 7.0.0-31.31
The failing dmesg identifies it as:
Linux version 7.0.0-31-generic
#31-Ubuntu SMP PREEMPT_DYNAMIC
Ubuntu 7.0.0-31.31-generic 7.0.14
Known-good kernel:
linux-image-6.17.0-42-generic 6.17.0-42.42+1
CPU:
AMD Ryzen 9 9950X3D
x86_64
48-bit physical address space
Motherboard/current DMI information:
Board vendor: Micro-Star International Co., Ltd.
Board name: PRO X870E-P WIFI (MS-7E70)
Board version: 2.0
BIOS vendor: American Megatrends International, LLC.
BIOS version: 2.A52
BIOS date: 06/29/2026
I did not save a separate DMI dump in the failing-boot artifact, so the BIOS
information above is from the current machine. I believe it is unchanged
from the failing test, but cannot prove that solely from the saved dmesg.
GPU configuration at the time of the failing 7.0 capture:
0000:01:00.0 NVIDIA GeForce RTX 5090 [10de:2b85]
0000:0d:00.0 NVIDIA GeForce RTX 3090 [10de:2204]
0000:11:00.0 NVIDIA GeForce RTX 3090 [10de:2204]
The RTX 5090 is attached directly under a separate root port. The two
RTX 3090s are sibling endpoints behind the affected chipset/switch path.
The machine now also contains an RTX 3060 behind 0000:0b:07.0. That GPU
was added after the failing 7.0 capture and is not part of the comparison
or arithmetic below. The current four-GPU configuration works on 6.17.
Relevant topology during the failing test
=========================================
0000:00:02.1 AMD GPP root port [1022:14db]
`-- 0000:03:00.0 AMD 600 Series switch upstream [1022:43f4]
`-- 0000:04:08.0 AMD 600 Series switch downstream [1022:43f5]
`-- 0000:0a:00.0 AMD 600 Series switch upstream [1022:43f4]
`-- 0000:0b
|-- 0000:0b:04.0 [bus 0d]
| |-- 0000:0d:00.0 RTX 3090 [10de:2204]
| `-- 0000:0d:00.1 NVIDIA HDA [10de:1aef]
|-- 0000:0b:05.0 [bus 0e] -- Realtek RTL8126
|-- 0000:0b:06.0 [bus 0f] -- Qualcomm WCN785x
|-- 0000:0b:08.0 [bus 11]
| |-- 0000:11:00.0 RTX 3090 [10de:2204]
| `-- 0000:11:00.1 NVIDIA HDA [10de:1aef]
|-- 0000:0b:0c.0 [bus 12] -- AMD USB
`-- 0000:0b:0d.0 [bus 13] -- AMD SATA
Failing test command line
=========================
The saved 7.0 boot used:
BOOT_IMAGE=/vmlinuz-7.0.0-31-generic
root=/dev/mapper/ubuntu--vg-ubuntu--lv
ro
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
pci=realloc=on
pci=resource_alignment=0000:00:02.1;0000:03:00.0;0000:04:08.0;0000:0a:00.0;0000:0b:04.0;0000:0b:08.0
The resource_alignment list was used to force reassignment of the affected
bridge path.
This is an important limitation of my existing comparison: the failing 7.0
capture used the targeted resource_alignment list, while the successful
6.17 boot used pci=realloc=on without that list. The 6.17 kernel still had
to reassign this hierarchy because the firmware-provided non-prefetchable
window conflicted with AMDIF031:00.
Prefetchable-window failure
===========================
Each RTX 3090 has:
BAR1: 0x800000000 = 32 GiB, prefetchable
BAR3: 0x02000000 = 32 MiB, prefetchable
Therefore each downstream bridge needs a composite prefetchable window of:
0x802000000 = 32 GiB + 32 MiB
The composite child window inherits 32 GiB alignment from BAR1.
On the failing 7.0 boot, the parent at 0000:0a:00.0 was assigned:
0000:0a:00.0:
[mem 0x1000000000-0x2003ffffff 64bit pref]
Its size is:
0x1004000000 = 64 GiB + 64 MiB
This is exactly the plain sum of two 32 GiB + 32 MiB child windows.
The first child was assigned successfully:
0000:0b:04.0:
[mem 0x1000000000-0x1801ffffff 64bit pref]
This consumes 32 GiB + 32 MiB. Its exclusive end is 0x1802000000.
Because the second child also requires 32 GiB alignment, it cannot start at
0x1802000000. Its next legal start is 0x2000000000. Placing another
0x802000000-byte child there requires an exclusive end of 0x2802000000.
Thus the required parent span is:
0x1000000000-0x2801ffffff
size 0x1802000000 = 96 GiB + 32 MiB
But Linux 7.0 allocated only:
size 0x1004000000 = 64 GiB + 64 MiB
The shortfall is:
0x7fe000000 = 32 GiB - 32 MiB
The kernel then reports:
pci 0000:0b:08.0: bridge window
[mem size 0x802000000 64bit pref]: can't assign; no space
pci 0000:0b:08.0: bridge window
[mem size 0x802000000 64bit pref]: failed to assign
This appears to be the same arithmetic as the original Vega20 report,
except that my composite windows contain a 32 GiB NVIDIA BAR1 plus a
32 MiB BAR3 rather than a 32 GiB AMD BAR plus a 2 MiB doorbell BAR.
On 6.17, the same hierarchy received:
0000:0a:00.0:
[mem 0xd000000000-0xe801ffffff 64bit pref]
That window is 0x1802000000 bytes, exactly 96 GiB + 32 MiB.
Both children then fit:
0000:0b:08.0:
[mem 0xd000000000-0xd801ffffff 64bit pref]
0000:0b:04.0:
[mem 0xe000000000-0xe801ffffff 64bit pref]
Both RTX 3090 BAR1 and BAR3 resources were assigned.
Non-prefetchable-window failure
===============================
The same boot also shows an analogous failure for the ordinary memory
window.
Each RTX 3090 branch contains approximately:
GPU BAR0: 16 MiB
GPU ROM: 512 KiB
HDA BAR0: 16 KiB
With bridge-window granularity, each child requests a 17 MiB ordinary
memory window. Its alignment is 16 MiB because of GPU BAR0.
Linux 7.0 assigned the nested parent at 0000:0a:00.0:
[mem 0xf2000000-0xf46fffff]
This is 39 MiB. The ordinary-memory child requirements below that parent
are 17 + 1 + 2 + 17 + 1 + 1 = 39 MiB, again the plain sum.
The two 17 MiB GPU child windows alone cannot be packed into 34 MiB because
each requires a 16 MiB-aligned start. If the first begins at offset zero,
it ends at 17 MiB. The second cannot begin until offset 32 MiB and ends at
49 MiB. The smaller resources may fill some of the intervening gap, but
they do not make the 39 MiB parent large enough.
The first GPU child succeeds:
pci 0000:0b:04.0: bridge window
[mem 0xf2000000-0xf30fffff]: assigned
The second fails:
pci 0000:0b:08.0: bridge window
[mem size 0x01100000]: can't assign; no space
pci 0000:0b:08.0: bridge window
[mem size 0x01100000]: failed to assign
On 6.17, reassignment produced:
0000:00:02.1:
[mem 0xf2000000-0xf59fffff] size 58 MiB
0000:0a:00.0:
[mem 0xf2000000-0xf57fffff] size 56 MiB
0000:0b:04.0:
[mem 0xf2000000-0xf37fffff] size 24 MiB
0000:0b:08.0:
[mem 0xf3800000-0xf4ffffff] size 24 MiB
Both GPU BAR0 resources were then assigned.
Endpoint result on Linux 7.0
============================
Once 0000:0b:08.0 loses both bridge windows, allocation failure cascades to
the endpoint. BAR0, BAR1, BAR3, the ROM, and the audio BAR all report
"can't assign; no space". The resulting sysfs resources for GPU BAR0,
BAR1, and BAR3 are all zero.
NVIDIA then refuses to probe the device because BAR0 is zero. Only the RTX
5090 and the first RTX 3090 were usable.
Known-good result
=================
With 6.17.0-42 and:
pci=realloc=on module_blacklist=nouveau
all relevant bridge windows and all physical BAR0, BAR1, and BAR3 resources
were assigned.
The system currently boots 6.17.0-42 and directly enumerates all four of
its current NVIDIA GPUs: RTX 5090, RTX 3090, RTX 3060, and RTX 3090.
Relationship to 3958bf16e2fe
=============================
The observed behavior seems consistent with the tail-side
multi-composite-resource problem described in this thread:
* pbus_size_mem() sizes the parent as the plain sum of child sizes;
* each composite child is larger than, and not an exact multiple of, its
effective alignment;
* assignment then inserts alignment padding between sibling children;
* the parent did not budget for that padding;
* 6.17 allocates a large enough parent while 7.0 does not.
The one-line max(r_size, align) -> ALIGN(r_size, align) change appears as if
it would cover this particular fixed topology, but I understand from the
discussion that it is not an acceptable general fix because it restores
unnecessary over-allocation and can regress systems with constrained
address space.
I have not verified the exact Ubuntu source against commit 3958bf16e2fe,
and I have not yet booted a kernel containing the newer proposed general
solution. Therefore I cannot yet prove the regression attribution by an
A/B test of the proposed patch.
Questions
=========
1. Does this look like the same multiple-composite-resource sizing
regression to you?
2. Would this NVIDIA/AMD-X870E topology be useful as another test case for
the newer series?
3. If so, which exact branch or patch-series revision should I test?
4. Is the intended solution expected to account for tail padding only when
multiple composite children are present, using the final calculated
alignment rather than unconditionally rounding every child to its own
alignment?
5. Once accepted upstream, is this fix expected to be marked for stable
backport to 7.0.y? Ubuntu 26.04 uses the 7.0 kernel series.
I have the complete failing 7.0 dmesg, complete successful 6.17 dmesg,
focused sysfs resource dumps, and current lspci output available. I can
provide sanitized copies or collect any additional read-only information
requested.
I can also test a candidate kernel, but this is a production multi-GPU
machine and I need to preserve the known-good 6.17 boot entry. Any test
would therefore be a one-time boot with a local-console and cold-power
recovery path rather than immediately replacing the default kernel.
Thanks,
JD
# Normalized excerpt from pre7-success-kernel.log and sysfs snapshots.
Linux version 6.17.0-42-generic
Command line: pci=realloc=on module_blacklist=nouveau
0000:00:02.1 Memory behind bridge: f2000000-f59fffff [size=58M]
0000:0b:04.0 Memory behind bridge: f2000000-f37fffff [size=24M]
0000:0b:08.0 Memory behind bridge: f3800000-f4ffffff [size=24M]
0000:11:00.0 BAR0: 0xf4000000-0xf4ffffff
0000:11:00.0 BAR1: 0xd000000000-0xd7ffffffff
0000:11:00.0 BAR3: 0xd800000000-0xd801ffffff
# Normalized excerpt from bridge-test-kernel.log (2026-09-11).
# Full historical log may be gitignored; this excerpt is deliberately tracked.
Linux version 7.0.0-31-generic
pci 0000:11:00.0: BAR 0 [mem size 0x01000000 64bit]: can't assign; no space
pci 0000:11:00.0: BAR 0 [mem size 0x01000000 64bit]: failed to assign
pci 0000:11:00.0: BAR 1 [mem size 0x800000000 64bit pref]: can't assign; no
space
pci 0000:11:00.0: BAR 3 [mem size 0x02000000 64bit pref]: failed to assign
0000:0b:08.0 Memory behind bridge: [disabled]
0000:11:00.0 resource BAR0: 0x0 0x0 0x0
0000:11:00.0 resource BAR1: 0x0 0x0 0x0
0000:11:00.0 resource BAR3: 0x0 0x0 0x0
