hello!
On Wed, Jun 17, 2026 at 09:49:49AM +1000, Jonathan Gray wrote:
> On Tue, Jun 16, 2026 at 09:00:08PM +0300, Petre Rodan wrote:
> >
> > hello Jonathan,
> >
> > On Tue, Jun 16, 2026 at 04:49:18PM +1000, Jonathan Gray wrote:
> > > On Mon, Jun 15, 2026 at 11:07:36PM +0300, Petre Rodan wrote:
> > > > On Mon, Jun 15, 2026 at 02:09:13PM +1000, Jonathan Gray wrote:
> > > > > On Sun, Jun 14, 2026 at 09:52:53PM +0300, Petre Rodan wrote:
> > > > > > 9:0:0: ATI Navi 21
> > > > > > 0x0000: Vendor ID: 1002, Product ID: 73bf
> > > > > > 0x0004: Command: 0006, Status: 0010
> > > > > > 0x0008: Class: 03 Display, Subclass: 00 VGA,
> > > > > > Interface: 00, Revision: c3
> > > > > > 0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
> > > > > > Cache Line Size: 10
> > > > > > 0x0010: BAR mem prefetchable 64bit addr:
> > > > > > 0x0000007800000000/0x400000000
> > > > > > 0x0018: BAR mem prefetchable 64bit addr:
> > > > > > 0x0000007c00000000/0x10000000
> > > > > > 0x0020: BAR io addr: 0x0000e000/0x0100
> > > > > > 0x0024: BAR mem 32bit addr: 0xfca00000/0x00100000
> > > > >
> > > > > framebuffer memory is described by pci bar 0x10
> > > > >
> > > > > the window size here is already 16GB, even without resizing the bar
> > > > >
> > > > > can you try this diff to force visible vram to 512MB?
> > > > >
> > > > > Index: sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c
> > > > > ===================================================================
> > > > > RCS file: /cvs/src/sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c,v
> > > > > diff -u -p -r1.20 amdgpu_gmc.c
> > > > > --- sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c 25 May 2026 08:56:32
> > > > > -0000 1.20
> > > > > +++ sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c 15 Jun 2026 03:53:34
> > > > > -0000
> > > > > @@ -216,6 +216,8 @@ void amdgpu_gmc_vram_location(struct amd
> > > > > uint64_t vis_limit = (uint64_t)amdgpu_vis_vram_limit << 20;
> > > > > uint64_t limit = (uint64_t)amdgpu_vram_limit << 20;
> > > > >
> > > > > + vis_limit = 512ULL << 20;
> > > > > +
> > > > > mc->vram_start = base;
> > > > > mc->vram_end = mc->vram_start + mc->mc_vram_size - 1;
> > > > > if (limit < mc->real_vram_size)
> > > >
> > > > I compared the amdgpu driver behaviour in OpenBSD vs Linux kernel
> > > > 6.18.35 and to me it looks like the errors start at this bsd specific
> > > > #ifdef block:
> > > >
> > > > RCS file: /cvs/src/sys/dev/pci/drm/amd/amdgpu/amdgpu_ttm.c,v
> > > > diff -u -p -r1.30 amdgpu_ttm.c
> > > > --- amdgpu_ttm.c 9 Mar 2026 23:57:53 -0000 1.30
> > > > +++ amdgpu_ttm.c 15 Jun 2026 19:00:39 -0000
> > > > @@ -2022,x +2022,y @@ int amdgpu_ttm_init(struct amdgpu_device
> > > > #else
> > > > if (bus_space_map(adev->memt, adev->gmc.aper_base,
> > > > adev->gmc.visible_vram_size,
> > > > BUS_SPACE_MAP_LINEAR | BUS_SPACE_MAP_PREFETCHABLE,
> > > > &adev->mman.aper_bsh)) {
> > > > adev->mman.aper_base_kaddr = NULL;
> > > > + printf("~~~ explicit NULL, visible_vram_size 0x%llX\n",
> > > > adev->gmc.visible_vram_size);
> > > > } else {
> > > > adev->mman.aper_base_kaddr = bus_space_vaddr(adev->memt,
> > > > adev->mman.aper_bsh);
> > > > + printf("~~~ aper_base_kaddr %p, visible_vram_size 0x%llX\n",
> > > > adev->mman.aper_base_kaddr,
> > > > + adev->gmc.visible_vram_size);
> > > > }
> > > > #endif
> > > >
> > > > <TLDR>
> > > > in OpenBSD if bus_space_map() is true then aper_base_kaddr is
> > > > explicitly NULLed out,
> > > > and for some reason this happens if visible_vram_size > 2GB.
> > > > in Linux aper_base_kaddr is a non-NULL pointer in all my tests.
> > > > </TLDR>
> > > >
> > > > # in OpenBSD with vis_limit between 512-2048MB, startx is happy to use
> > > > the modsetting driver:
> > > >
> > > > VRAM: 16368M 0x0000008000000000 - 0x00000083FEFFFFFF (16368M used)
> > > > GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
> > > > ~~~ aper_base_kaddr 0xffff8000699ea000, visible_vram_size 0x80000000
> > > > amdgpu0: SIENNA_CICHLID GC 10.3.0 60 CU rev 0x01
> > > > amdgpu0: 1920x1080, 32bpp
> > > > wsdisplay0 at amdgpu0 mux 1
> > > >
> > > >
> > > > # in OpenBSD with vis_limit >= 2560UL<<20, aper_base_kaddr is
> > > > explicitly set to NULL and the init failure chain happens:
> > > >
> > > > VRAM: 16368M 0x0000008000000000 - 0x00000083FEFFFFFF (16368M used)
> > > > GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
> > > > ~~~ explicit NULL, visible_vram_size 0xA0000000
> > > > [drm] *ERROR* v11 visible_vram_size a0000000 or aper_base_kaddr 0x0 is
> > > > not initialized.
> > > > drm:pid0:psp_sw_init *ERROR* Failed to process memory training!
> > > >
> > > > drm:pid0:amdgpu_device_ip_init *ERROR* sw_init of IP block <psp> failed
> > > > -22
> > > > drm:pid0:amdgpu_device_init *ERROR* amdgpu_device_ip_init failed
> > > >
> > > > drm:pid0:amdgpu_driver_load_kms *ERROR* Fatal error during GPU init
> > > > WARNING !amdgpu_irq_enabled(adev, src, type) failed at
> > > > /sys/dev/pci/drm/amd/amdgpu/amdgpu_irq.c:650
> > > >
> > > >
> > > > # in linux, even with the default unlimited vis_limit, aper_base_kaddr
> > > > does not get NULLed out:
> > > >
> > > > [ 8.101797] amdgpu 0000:09:00.0: amdgpu: VRAM: 16368M
> > > > 0x0000008000000000 - 0x00000083FEFFFFFF (16368M used)
> > > > [ 8.101799] amdgpu 0000:09:00.0: amdgpu: GART: 512M
> > > > 0x0000000000000000 - 0x000000001FFFFFFF
> > > > [ 8.101809] [drm] Detected VRAM RAM=16368M, BAR=16384M
> > > > [ 8.101810] [drm] RAM width 256bits GDDR6
> > > > [ 8.101908] amdgpu: ~~~ aper_base_kaddr 00000000dbe1887b,
> > > > visible_vram_size 0x3FF000000
> > > > [ 8.101937] amdgpu 0000:09:00.0: amdgpu: amdgpu: 16368M of VRAM
> > > > memory ready
> > > > [ 8.101939] amdgpu 0000:09:00.0: amdgpu: amdgpu: 16002M of GTT
> > > > memory ready.
> > > >
> > > > am I getting warmer?
> > >
> > > My 8GB Vega 56 shows up with a 256M window so I can't test on hardware.
> > > Can you try the below diff and report back the dmesg changes?
> >
> > my pleasure. thank you for taking your time.
> >
> > > 33:0:0: ATI Radeon Rx Vega
> > > 0x0000: Vendor ID: 1002, Product ID: 687f
> > > 0x0004: Command: 0007, Status: 0010
> > > 0x0008: Class: 03 Display, Subclass: 00 VGA,
> > > Interface: 00, Revision: c3
> > > 0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
> > > Cache Line Size: 10
> > > 0x0010: BAR mem prefetchable 64bit addr: 0x00000000e0000000/0x10000000
> > > 0x0018: BAR mem prefetchable 64bit addr: 0x00000000f0000000/0x00200000
> > > 0x0020: BAR io addr: 0x0000e000/0x0100
> > > 0x0024: BAR mem 32bit addr: 0xfcb00000/0x00080000
> > >
> > > Index: sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c
> > > ===================================================================
> > > RCS file: /cvs/src/sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c,v
> > > diff -u -p -r1.73 amdgpu_drv.c
> > > --- sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c 15 Jun 2026 02:46:12
> > > -0000 1.73
> > > +++ sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c 16 Jun 2026 05:29:53
> > > -0000
> > > @@ -3409,6 +3409,8 @@ amdgpu_attach(struct device *parent, str
> > > printf(": can't get framebuffer info\n");
> > > return;
> > > }
> > > +printf("\n%s pci_mapreg_info fb_aper_offset 0x%lx fb_aper_size 0x%lx\n",
> > > + __func__, adev->fb_aper_offset, adev->fb_aper_size);
> > >
> > > if (adev->fb_aper_offset == 0) {
> > > bus_size_t start, end, pci_mem_end;
> > > @@ -3432,6 +3434,8 @@ amdgpu_attach(struct device *parent, str
> > > pci_conf_write(pa->pa_pc, pa->pa_tag,
> > > AMDGPU_PCI_MEM + 4, (uint64_t)base >> 32);
> > > adev->fb_aper_offset = base;
> > > +printf("\n%s fallback fb_aper_offset 0x%lx fb_aper_size 0x%lx\n",
> > > + __func__, adev->fb_aper_offset, adev->fb_aper_size);
> > > }
> > >
> > > if (adev->family >= CHIP_BONAIRE)
> > > Index: sys/arch/amd64/amd64/bus_space.c
> > > ===================================================================
> > > RCS file: /cvs/src/sys/arch/amd64/amd64/bus_space.c,v
> > > diff -u -p -r1.31 bus_space.c
> > > --- sys/arch/amd64/amd64/bus_space.c 4 Jun 2026 05:22:04 -0000
> > > 1.31
> > > +++ sys/arch/amd64/amd64/bus_space.c 16 Jun 2026 05:45:39 -0000
> > > @@ -381,8 +381,10 @@ bus_space_map(bus_space_tag_t t, bus_add
> > > */
> > > error = extent_alloc_region(ex, bpa, size,
> > > EX_NOWAIT | (ioport_malloc_safe ? EX_MALLOCOK : 0));
> > > - if (error)
> > > + if (error) {
> > > +printf("\n%s extent_alloc_region error %d\n", __func__, error);
> > > return (error);
> > > + }
> > >
> > > /*
> > > * For I/O space, that's all she wrote.
> > > @@ -414,6 +416,7 @@ bus_space_map(bus_space_tag_t t, bus_add
> > > bpa, size);
> > > printf("bus_space_map: can't free region\n");
> > > }
> > > +printf("\n%s x86_mem_add_mapping error %d\n", __func__, error);
> > > }
> > >
> > > return (error);
> > > @@ -522,8 +525,10 @@ x86_mem_add_mapping(bus_addr_t bpa, bus_
> > > map_size = endpa - pa;
> > >
> > > va = (vaddr_t)km_alloc(map_size, &kv_any, &kp_none, &kd_nowait);
> > > - if (va == 0)
> > > + if (va == 0) {
> > > +printf("\n%s km_alloc ENOMEM\n", __func__);
> > > return (ENOMEM);
> > > + }
> > >
> > > *bshp = (bus_space_handle_t)(va + (bpa & PGOFSET));
> >
> > I took the liberty of adding a few more printfs in order to dig a bit
> > deeper.
> > and I got stuck in uvm_addr_invoke(), which calls an unknown to me
> > uaddr_select function via a pointer.
> > the particular pointer address being used does not seem to belong to the
> > _select functions within uvm_addr.c however - see the dmesg outputs. so
> > maybe there is a place where &kv_any's _select function used was created
> > from scratch?
> >
> > attached you'll find 3 files:
> > - current patch against /sys OPENBSD_7_9 for printf context
> > - dmesg with vis_limit 4GB, resizebar and above 4G decoding enabled in bios
> > - dmesg with vis_limit 4GB, resizebar and above 4G decoding disabled in bios
> >
> > I guess you're no longer interested in the 2GB vis_limit since it will not
> > trigger any additional err messages.
> > also I see no notable changes if I enable/disable those 2 options in bios,
> > I will leave them on from now on.
> >
> > I am used to write code for 16bit microcontrollers so please allow me to
> > accommodate a bit with this virtual memory mapping madness :)
>
> Thanks. The problem is the size of the kernel virtual address space
> on amd64.
>
> #define VM_MIN_KERNEL_ADDRESS 0xffff800000000000
> #define VM_MAX_KERNEL_ADDRESS 0xffff800100000000
>
> Is 4GB.
>
> This should be increased, but that may require further changes in the
> part of the kernel involved in memory mapping (the pmap).
I see.
16GB video cards are common these days.
and linux amdgpu supports a few 32, 48GB cards already (Radeon Pro W6800,
Radeon PRO W7000)
> So as a first step, I will commit a change to limit the visible vram
> amdgpu is using. Something like the amdgpu_gmc_vram_location() patch
> you previously tested.
thank you.
please do ping me if you want me to test amdgpu patches once the vm headroom
change happens.
best regards,
peter
