On Fri, Apr 20, 2018 at 04:59:27AM +0200, Mark Kettenis wrote:
> > Date: Fri, 20 Apr 2018 11:41:10 +1000
> > From: Jonathan Matthew <jonat...@d14n.org>
> >
> > We just got some R6415s with 512GB of memory and found it's hard to boot
> > them
> > because the kernel message buffer ends up past the 512GB boundary, but the
> > direct map is "only" 512GB, so initmsgbuf() crashes.
> >
> > I found 'mach mem =1G' moves the buffer below 512GB (but 'mach mem
> > =2G' doesn't!), which let me write the diff below that clips memory
> > clusters at 512GB. This means uvm doesn't know about any memory
> > past 512GB and the code in init_x86_64() that steals memory for
> > buffers doesn't use it either.
> >
> > With this the machine boots and I still have almost all of the memory:
> >
> > OpenBSD 6.3-current (GENERIC.MP) #3: Fri Apr 20 07:51:06 AEST 2018
> >
> > jonat...@r6415.embarrassm.net:/home/jonathan/src/sys/arch/amd64/compile/GENERIC.MP
> > real mem = 547253186560 (521901MB)
> > avail mem = 530660573184 (506077MB)
> >
> > The memory layout looks like this:
> >
> > boot> mach mem
> > Region 0: type 1 at 0x0 for 572KB
> > Region 1: type 4 at 0x8f000 for 4KB
> > Region 2: type 1 at 0x90000 for 64KB
> > Region 3: type 1 at 0x100000 for 1722364KB
> > Region 4: type 2 at 0x692ff000 for 37888KB
> > Region 5: type 4 at 0x6b7ff000 for 14528KB
> > Region 6: type 3 at 0x6c62f000 for 2048KB
> > Region 7: type 1 at 0x6c82f000 for 57156KB
> > Region 8: type 1 at 0x100000000 for 132107776KB
> > Region 9: type 1 at 0x2080000000 for 134217216KB
> > Region 10: type 1 at 0x4080000000 for 134217216KB
> > Region 11: type 1 at 0x6080000000 for 134217216KB
> > Region 12: type 2 at 0x70000000 for 524288KB
> > Region 13: type 2 at 0xfec10000 for 4KB
> > Region 14: type 2 at 0xfed80000 for 4KB
> > Region 15: type 2 at 0x207f380000 for 12800KB
> > Region 16: type 2 at 0x407ff80000 for 512KB
> > Region 17: type 2 at 0x607ff80000 for 512KB
> > Region 18: type 2 at 0x807ff80000 for 512KB
> > Low ram: 640KB High ram: 2358272KB
> > Total free memory: 536539580KB
> >
> >
> > ok? or should we look at expanding the direct map instead?
>
> I'd say we should do something like this regardless. However...
>
> > Index: machdep.c
> > ===================================================================
> > RCS file: /cvs/src/sys/arch/amd64/amd64/machdep.c,v
> > retrieving revision 1.241
> > diff -u -p -u -p -r1.241 machdep.c
> > --- machdep.c 12 Apr 2018 17:13:43 -0000 1.241
> > +++ machdep.c 19 Apr 2018 23:32:23 -0000
> > @@ -1439,6 +1439,16 @@ init_x86_64(paddr_t first_avail)
> > continue;
> > }
> >
> > + /*
> > + * The direct map is limited to 512GB of memory, so
> > + * discard anything above that.
> > + */
> > + if (e1 >= (uint64_t)512*1024*1024*1024) {
> > + e1 = ((uint64_t)512*1024*1024*1024) - PAGE_SIZE;
>
> Why are you subtracting PAGE_SIZE here?
Mostly because I didn't read the code closely enough. I thought that was
supposed to be the address of the last valid page, but it looks like it's
the next one. Not subtracting PAGE_SIZE here also works.
>
> > + if (s1 > e1)
> > + continue;
> > + }
> > +
> > /* Crop stuff into "640K hole" */
> > if (s1 < IOM_BEGIN && e1 > IOM_BEGIN)
> > e1 = IOM_BEGIN;
> >
> >
