On Wed, Jun 17, 2026 at 06:25:04AM +0300, Petre Rodan wrote:
>
> hello!
>
> On Wed, Jun 17, 2026 at 09:49:49AM +1000, Jonathan Gray wrote:
> > On Tue, Jun 16, 2026 at 09:00:08PM +0300, Petre Rodan wrote:
> > >
> > > hello Jonathan,
> > >
> > > On Tue, Jun 16, 2026 at 04:49:18PM +1000, Jonathan Gray wrote:
> > > > On Mon, Jun 15, 2026 at 11:07:36PM +0300, Petre Rodan wrote:
> > > > > On Mon, Jun 15, 2026 at 02:09:13PM +1000, Jonathan Gray wrote:
> > > > > > On Sun, Jun 14, 2026 at 09:52:53PM +0300, Petre Rodan wrote:
> > > > > > > 9:0:0: ATI Navi 21
> > > > > > > 0x0000: Vendor ID: 1002, Product ID: 73bf
> > > > > > > 0x0004: Command: 0006, Status: 0010
> > > > > > > 0x0008: Class: 03 Display, Subclass: 00 VGA,
> > > > > > > Interface: 00, Revision: c3
> > > > > > > 0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
> > > > > > > Cache Line Size: 10
> > > > > > > 0x0010: BAR mem prefetchable 64bit addr:
> > > > > > > 0x0000007800000000/0x400000000
> > > > > > > 0x0018: BAR mem prefetchable 64bit addr:
> > > > > > > 0x0000007c00000000/0x10000000
> > > > > > > 0x0020: BAR io addr: 0x0000e000/0x0100
> > > > > > > 0x0024: BAR mem 32bit addr: 0xfca00000/0x00100000
> > > > > >
> > > > > > framebuffer memory is described by pci bar 0x10
> > > > > >
> > > > > > the window size here is already 16GB, even without resizing the bar
> > > > > >
> > > > > > can you try this diff to force visible vram to 512MB?
> > > > > >
> > > > > > Index: sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c
> > > > > > ===================================================================
> > > > > > RCS file: /cvs/src/sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c,v
> > > > > > diff -u -p -r1.20 amdgpu_gmc.c
> > > > > > --- sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c 25 May 2026 08:56:32
> > > > > > -0000 1.20
> > > > > > +++ sys/dev/pci/drm/amd/amdgpu/amdgpu_gmc.c 15 Jun 2026 03:53:34
> > > > > > -0000
> > > > > > @@ -216,6 +216,8 @@ void amdgpu_gmc_vram_location(struct amd
> > > > > > uint64_t vis_limit = (uint64_t)amdgpu_vis_vram_limit << 20;
> > > > > > uint64_t limit = (uint64_t)amdgpu_vram_limit << 20;
> > > > > >
> > > > > > + vis_limit = 512ULL << 20;
> > > > > > +
> > > > > > mc->vram_start = base;
> > > > > > mc->vram_end = mc->vram_start + mc->mc_vram_size - 1;
> > > > > > if (limit < mc->real_vram_size)
> > > > >
> > > > > I compared the amdgpu driver behaviour in OpenBSD vs Linux kernel
> > > > > 6.18.35 and to me it looks like the errors start at this bsd specific
> > > > > #ifdef block:
> > > > >
> > > > > RCS file: /cvs/src/sys/dev/pci/drm/amd/amdgpu/amdgpu_ttm.c,v
> > > > > diff -u -p -r1.30 amdgpu_ttm.c
> > > > > --- amdgpu_ttm.c 9 Mar 2026 23:57:53 -0000 1.30
> > > > > +++ amdgpu_ttm.c 15 Jun 2026 19:00:39 -0000
> > > > > @@ -2022,x +2022,y @@ int amdgpu_ttm_init(struct amdgpu_device
> > > > > #else
> > > > > if (bus_space_map(adev->memt, adev->gmc.aper_base,
> > > > > adev->gmc.visible_vram_size,
> > > > > BUS_SPACE_MAP_LINEAR | BUS_SPACE_MAP_PREFETCHABLE,
> > > > > &adev->mman.aper_bsh)) {
> > > > > adev->mman.aper_base_kaddr = NULL;
> > > > > + printf("~~~ explicit NULL, visible_vram_size 0x%llX\n",
> > > > > adev->gmc.visible_vram_size);
> > > > > } else {
> > > > > adev->mman.aper_base_kaddr = bus_space_vaddr(adev->memt,
> > > > > adev->mman.aper_bsh);
> > > > > + printf("~~~ aper_base_kaddr %p, visible_vram_size 0x%llX\n",
> > > > > adev->mman.aper_base_kaddr,
> > > > > + adev->gmc.visible_vram_size);
> > > > > }
> > > > > #endif
> > > > >
> > > > > <TLDR>
> > > > > in OpenBSD if bus_space_map() is true then aper_base_kaddr is
> > > > > explicitly NULLed out,
> > > > > and for some reason this happens if visible_vram_size > 2GB.
> > > > > in Linux aper_base_kaddr is a non-NULL pointer in all my tests.
> > > > > </TLDR>
> > > > >
> > > > > # in OpenBSD with vis_limit between 512-2048MB, startx is happy to
> > > > > use the modsetting driver:
> > > > >
> > > > > VRAM: 16368M 0x0000008000000000 - 0x00000083FEFFFFFF (16368M used)
> > > > > GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
> > > > > ~~~ aper_base_kaddr 0xffff8000699ea000, visible_vram_size 0x80000000
> > > > > amdgpu0: SIENNA_CICHLID GC 10.3.0 60 CU rev 0x01
> > > > >
> > > > > amdgpu0: 1920x1080, 32bpp
> > > > > wsdisplay0 at amdgpu0 mux 1
> > > > >
> > > > >
> > > > > # in OpenBSD with vis_limit >= 2560UL<<20, aper_base_kaddr is
> > > > > explicitly set to NULL and the init failure chain happens:
> > > > >
> > > > > VRAM: 16368M 0x0000008000000000 - 0x00000083FEFFFFFF (16368M used)
> > > > > GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
> > > > > ~~~ explicit NULL, visible_vram_size 0xA0000000
> > > > > [drm] *ERROR* v11 visible_vram_size a0000000 or aper_base_kaddr 0x0
> > > > > is not initialized.
> > > > > drm:pid0:psp_sw_init *ERROR* Failed to process memory training!
> > > > >
> > > > > drm:pid0:amdgpu_device_ip_init *ERROR* sw_init of IP block <psp>
> > > > > failed -22
> > > > > drm:pid0:amdgpu_device_init *ERROR* amdgpu_device_ip_init failed
> > > > >
> > > > > drm:pid0:amdgpu_driver_load_kms *ERROR* Fatal error during GPU init
> > > > > WARNING !amdgpu_irq_enabled(adev, src, type) failed at
> > > > > /sys/dev/pci/drm/amd/amdgpu/amdgpu_irq.c:650
> > > > >
> > > > >
> > > > > # in linux, even with the default unlimited vis_limit,
> > > > > aper_base_kaddr does not get NULLed out:
> > > > >
> > > > > [ 8.101797] amdgpu 0000:09:00.0: amdgpu: VRAM: 16368M
> > > > > 0x0000008000000000 - 0x00000083FEFFFFFF (16368M used)
> > > > > [ 8.101799] amdgpu 0000:09:00.0: amdgpu: GART: 512M
> > > > > 0x0000000000000000 - 0x000000001FFFFFFF
> > > > > [ 8.101809] [drm] Detected VRAM RAM=16368M, BAR=16384M
> > > > > [ 8.101810] [drm] RAM width 256bits GDDR6
> > > > > [ 8.101908] amdgpu: ~~~ aper_base_kaddr 00000000dbe1887b,
> > > > > visible_vram_size 0x3FF000000
> > > > > [ 8.101937] amdgpu 0000:09:00.0: amdgpu: amdgpu: 16368M of VRAM
> > > > > memory ready
> > > > > [ 8.101939] amdgpu 0000:09:00.0: amdgpu: amdgpu: 16002M of GTT
> > > > > memory ready.
> > > > >
> > > > > am I getting warmer?
> > > >
> > > > My 8GB Vega 56 shows up with a 256M window so I can't test on hardware.
> > > > Can you try the below diff and report back the dmesg changes?
> > >
> > > my pleasure. thank you for taking your time.
> > >
> > > > 33:0:0: ATI Radeon Rx Vega
> > > > 0x0000: Vendor ID: 1002, Product ID: 687f
> > > > 0x0004: Command: 0007, Status: 0010
> > > > 0x0008: Class: 03 Display, Subclass: 00 VGA,
> > > > Interface: 00, Revision: c3
> > > > 0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
> > > > Cache Line Size: 10
> > > > 0x0010: BAR mem prefetchable 64bit addr:
> > > > 0x00000000e0000000/0x10000000
> > > > 0x0018: BAR mem prefetchable 64bit addr:
> > > > 0x00000000f0000000/0x00200000
> > > > 0x0020: BAR io addr: 0x0000e000/0x0100
> > > > 0x0024: BAR mem 32bit addr: 0xfcb00000/0x00080000
> > > >
> > > > Index: sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c
> > > > ===================================================================
> > > > RCS file: /cvs/src/sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c,v
> > > > diff -u -p -r1.73 amdgpu_drv.c
> > > > --- sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c 15 Jun 2026 02:46:12
> > > > -0000 1.73
> > > > +++ sys/dev/pci/drm/amd/amdgpu/amdgpu_drv.c 16 Jun 2026 05:29:53
> > > > -0000
> > > > @@ -3409,6 +3409,8 @@ amdgpu_attach(struct device *parent, str
> > > > printf(": can't get framebuffer info\n");
> > > > return;
> > > > }
> > > > +printf("\n%s pci_mapreg_info fb_aper_offset 0x%lx fb_aper_size
> > > > 0x%lx\n",
> > > > + __func__, adev->fb_aper_offset, adev->fb_aper_size);
> > > >
> > > > if (adev->fb_aper_offset == 0) {
> > > > bus_size_t start, end, pci_mem_end;
> > > > @@ -3432,6 +3434,8 @@ amdgpu_attach(struct device *parent, str
> > > > pci_conf_write(pa->pa_pc, pa->pa_tag,
> > > > AMDGPU_PCI_MEM + 4, (uint64_t)base >> 32);
> > > > adev->fb_aper_offset = base;
> > > > +printf("\n%s fallback fb_aper_offset 0x%lx fb_aper_size 0x%lx\n",
> > > > + __func__, adev->fb_aper_offset, adev->fb_aper_size);
> > > > }
> > > >
> > > > if (adev->family >= CHIP_BONAIRE)
> > > > Index: sys/arch/amd64/amd64/bus_space.c
> > > > ===================================================================
> > > > RCS file: /cvs/src/sys/arch/amd64/amd64/bus_space.c,v
> > > > diff -u -p -r1.31 bus_space.c
> > > > --- sys/arch/amd64/amd64/bus_space.c 4 Jun 2026 05:22:04 -0000
> > > > 1.31
> > > > +++ sys/arch/amd64/amd64/bus_space.c 16 Jun 2026 05:45:39 -0000
> > > > @@ -381,8 +381,10 @@ bus_space_map(bus_space_tag_t t, bus_add
> > > > */
> > > > error = extent_alloc_region(ex, bpa, size,
> > > > EX_NOWAIT | (ioport_malloc_safe ? EX_MALLOCOK : 0));
> > > > - if (error)
> > > > + if (error) {
> > > > +printf("\n%s extent_alloc_region error %d\n", __func__, error);
> > > > return (error);
> > > > + }
> > > >
> > > > /*
> > > > * For I/O space, that's all she wrote.
> > > > @@ -414,6 +416,7 @@ bus_space_map(bus_space_tag_t t, bus_add
> > > > bpa, size);
> > > > printf("bus_space_map: can't free region\n");
> > > > }
> > > > +printf("\n%s x86_mem_add_mapping error %d\n", __func__, error);
> > > > }
> > > >
> > > > return (error);
> > > > @@ -522,8 +525,10 @@ x86_mem_add_mapping(bus_addr_t bpa, bus_
> > > > map_size = endpa - pa;
> > > >
> > > > va = (vaddr_t)km_alloc(map_size, &kv_any, &kp_none, &kd_nowait);
> > > > - if (va == 0)
> > > > + if (va == 0) {
> > > > +printf("\n%s km_alloc ENOMEM\n", __func__);
> > > > return (ENOMEM);
> > > > + }
> > > >
> > > > *bshp = (bus_space_handle_t)(va + (bpa & PGOFSET));
> > >
> > > I took the liberty of adding a few more printfs in order to dig a bit
> > > deeper.
> > > and I got stuck in uvm_addr_invoke(), which calls an unknown to me
> > > uaddr_select function via a pointer.
> > > the particular pointer address being used does not seem to belong to the
> > > _select functions within uvm_addr.c however - see the dmesg outputs. so
> > > maybe there is a place where &kv_any's _select function used was created
> > > from scratch?
> > >
> > > attached you'll find 3 files:
> > > - current patch against /sys OPENBSD_7_9 for printf context
> > > - dmesg with vis_limit 4GB, resizebar and above 4G decoding enabled in
> > > bios
> > > - dmesg with vis_limit 4GB, resizebar and above 4G decoding disabled in
> > > bios
> > >
> > > I guess you're no longer interested in the 2GB vis_limit since it will
> > > not trigger any additional err messages.
> > > also I see no notable changes if I enable/disable those 2 options in
> > > bios, I will leave them on from now on.
> > >
> > > I am used to write code for 16bit microcontrollers so please allow me to
> > > accommodate a bit with this virtual memory mapping madness :)
> >
> > Thanks. The problem is the size of the kernel virtual address space
> > on amd64.
> >
> > #define VM_MIN_KERNEL_ADDRESS 0xffff800000000000
> > #define VM_MAX_KERNEL_ADDRESS 0xffff800100000000
> >
> > Is 4GB.
> >
> > This should be increased, but that may require further changes in the
> > part of the kernel involved in memory mapping (the pmap).
>
> I see.
> 16GB video cards are common these days.
> and linux amdgpu supports a few 32, 48GB cards already (Radeon Pro W6800,
> Radeon PRO W7000)
>
> > So as a first step, I will commit a change to limit the visible vram
> > amdgpu is using. Something like the amdgpu_gmc_vram_location() patch
> > you previously tested.
>
> thank you.
> please do ping me if you want me to test amdgpu patches once the vm headroom
> change happens.
>
> best regards,
> peter
You could try the following to increase the KVA size to 64GB. Other
changes may be required. I'm not familiar with amd64 pmap details.
Index: sys/arch/amd64/include/vmparam.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/include/vmparam.h,v
diff -u -p -r1.25 vmparam.h
--- sys/arch/amd64/include/vmparam.h 8 Nov 2024 01:44:52 -0000 1.25
+++ sys/arch/amd64/include/vmparam.h 17 Jun 2026 02:55:20 -0000
@@ -98,7 +98,7 @@
#define VM_MIN_STACK_ADDRESS 0x0000600000000000
#endif
#define VM_MIN_KERNEL_ADDRESS 0xffff800000000000
-#define VM_MAX_KERNEL_ADDRESS 0xffff800100000000
+#define VM_MAX_KERNEL_ADDRESS 0xffff801000000000
/* map PIE into approximately the first quarter of user va space */
#define VM_PIE_MIN_ADDR VM_MIN_ADDRESS
