Re: panic: UMA: Increase vm.boot_pages on Dell R920 r279210
On Sat, May 2, 2015 at 10:25 PM, Adrian Chadd adr...@freebsd.org wrote: hi, Hm, should we be upping this limit automatically? Can we get cpu counts or memory amount early enough in boot to have a hope of auto-tuning? 64 seems low, 1024 seems high as a default. :) What is it that's exhausting the boot_pages? I'm semi-guessing it's the number of vm radix tree nodes needed for the TiB of memory. The only thing I'm aware of (allow for ignorance here) that consumes boot_pages and scales with the cpu count is the uma zone used for uma cache objects, but on amd64 this zone only needs 640 + cpus * 128 bytes, or about 4 pages for 120 cpus. vm radix nodes are 144 bytes each on amd64, and by my back-of-the-envelope calculations (using traces of non-vm-radix boot_page use from another amd64 system), 64 boot_pages would be exhausted after about 1000 vm radix nodes were allocated. It would be interesting to know how many boot_pages were actually required for this particular system. In any event, since startup_alloc() is designed to exhaust all the boot_pages before switching to the normal allocators, it doesn't seem necessarily harmful to err on the high side either in bumping up the static default or introducing an auto-tuned value (provided the excess is not so perversely large that startup_alloc() isn't able to make use of an embarrassment of pages due to zone creation timing and usage patterns). We know the number of cpus at the time boot_pages is put to use, but I don't think we know how much memory there is (and even less sure that even if we did, we'd really want to try to estimate things the vm radix tree size in a generic way). Something like a default of boot_pages = max(64, 32 + k * cpus) might be sufficient for k = 4 or 8 (gathering some data points would give a clue here), and palatable since it is at a minimum the current value that's been in use, and at the other end approaches a modest commitment of 16 or 32 KiB per cpu in the worst case (unused and unreclaimed boot_pages with high cpu count). -Patrick On 24 March 2015 at 13:00, Keith White kwh...@site.uottawa.ca wrote: On Tue, 24 Mar 2015, Rui Paulo wrote: On Mar 24, 2015, at 04:19, kwh...@site.uottawa.ca wrote: I'm using /boot/loader.conf. Is there another place I should be doing this? No, that's correct, but apparently there's a problem: the RDTUN sysctl is not picked up early enough. Can you try this patch? I haven't really tested it. :-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 79665ba..a764788 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -134,8 +134,9 @@ long first_page; int vm_page_zero_count; static int boot_pages = UMA_BOOT_PAGES; -SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN, boot_pages, 0, - number of pages allocated for bootstrapping the VM system); +SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, +boot_pages, 0, +number of pages allocated for bootstrapping the VM system); static int pa_tryrelock_restart; SYSCTL_INT(_vm, OID_AUTO, tryrelock_restart, CTLFLAG_RD, @@ -349,6 +350,7 @@ vm_page_startup(vm_offset_t vaddr) * Allocate memory for use when boot strapping the kernel memory * allocator. */ + TUNABLE_INT_FETCH(vm.boot_pages, boot_pages); new_end = end - (boot_pages * UMA_SLAB_SIZE); new_end = trunc_page(new_end); mapped = pmap_map(vaddr, new_end, end, @@ -443,7 +445,7 @@ vm_page_startup(vm_offset_t vaddr) -- Rui Paulo Patch tried. Success! I now get this after setting vm.boot_pages=1024 in /boot/loader.conf: Booting... GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #1: Tue Mar 24 13:44:48 UTC 2015 root@:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.5.1 (tags/RELEASE_351/final 225668) 20150115 WARNING: WITNESS option enabled, expect reduced performance. UMA startup boot_pages: 1024 ... And can start all 120 processors. Thanks! ...keith -- Keith White, genie.uottawa.ca engineering.uottawa.ca kwh...@uottawa.ca [+1 613 562 5800 x6681] ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___
Re: panic: UMA: Increase vm.boot_pages on Dell R920 r279210
hi, Hm, should we be upping this limit automatically? Can we get cpu counts or memory amount early enough in boot to have a hope of auto-tuning? 64 seems low, 1024 seems high as a default. :) -adrian On 24 March 2015 at 13:00, Keith White kwh...@site.uottawa.ca wrote: On Tue, 24 Mar 2015, Rui Paulo wrote: On Mar 24, 2015, at 04:19, kwh...@site.uottawa.ca wrote: I'm using /boot/loader.conf. Is there another place I should be doing this? No, that's correct, but apparently there's a problem: the RDTUN sysctl is not picked up early enough. Can you try this patch? I haven't really tested it. :-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 79665ba..a764788 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -134,8 +134,9 @@ long first_page; int vm_page_zero_count; static int boot_pages = UMA_BOOT_PAGES; -SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN, boot_pages, 0, - number of pages allocated for bootstrapping the VM system); +SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, +boot_pages, 0, +number of pages allocated for bootstrapping the VM system); static int pa_tryrelock_restart; SYSCTL_INT(_vm, OID_AUTO, tryrelock_restart, CTLFLAG_RD, @@ -349,6 +350,7 @@ vm_page_startup(vm_offset_t vaddr) * Allocate memory for use when boot strapping the kernel memory * allocator. */ + TUNABLE_INT_FETCH(vm.boot_pages, boot_pages); new_end = end - (boot_pages * UMA_SLAB_SIZE); new_end = trunc_page(new_end); mapped = pmap_map(vaddr, new_end, end, @@ -443,7 +445,7 @@ vm_page_startup(vm_offset_t vaddr) -- Rui Paulo Patch tried. Success! I now get this after setting vm.boot_pages=1024 in /boot/loader.conf: Booting... GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #1: Tue Mar 24 13:44:48 UTC 2015 root@:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.5.1 (tags/RELEASE_351/final 225668) 20150115 WARNING: WITNESS option enabled, expect reduced performance. UMA startup boot_pages: 1024 ... And can start all 120 processors. Thanks! ...keith -- Keith White, genie.uottawa.ca engineering.uottawa.ca kwh...@uottawa.ca [+1 613 562 5800 x6681] ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: UMA: Increase vm.boot_pages on Dell R920 r279210
On Mar 19, 2015, at 07:34, Keith White kwh...@site.uottawa.ca wrote: I tried the suggestion Increase vm.boot_pages but am unsure how much. I tried doubling to 128, and even excessive(?) values like 102400; but got the same panic. How are you trying to change the value? Can you build a custom kernel and confirm the value in uma_startup()? -- Rui Paulo - I'm using /boot/loader.conf. Is there another place I should be doing this? vfs.mountroot.timeout=10 boot_multicons=YES boot_serial=YES comconsole_speed=115200 console=comconsole,vidconsole vm.boot_pages=1024 hw.mfi.mrsas_enable=1 - With a custom kernel, the value printed is 64 not my expected 1024: ... Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0: Tue Mar 24 04:55:47 UTC 2015 root@:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.5.1 (tags/RELEASE_351/final 225668) 20150115 WARNING: WITNESS option enabled, expect reduced performance. === UMA startup boot_pages: 64 === VT: running with driver vga. CPU: Intel(R) Xeon(R) CPU E7-4870 v2 @ 2.30GHz (2300.05-MHz K8-class CPU) Origin=GenuineIntel Id=0x306e7 Family=0x6 Model=0x3e Stepping=7 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0x7fbee3ffSSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND AMD Features=0x2c100800SYSCALL,NX,Page1GB,RDTSCP,LM AMD Features2=0x1LAHF Structured Extended Features=0x281FSGSBASE,SMEP,ERMS XSAVE Features=0x1XSAVEOPT VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 1099478073344 (1048544 MB) avail memory = 1069175943168 (1019645 MB) Event timer LAPIC quality 600 ACPI APIC Table: DELL PE_SC3 FreeBSD/SMP: Multiprocessor System Detected: 80 CPUs FreeBSD/SMP: 4 package(s) x 10 core(s) x 2 SMT threads ... -- After boot, the value is as expected: # sysctl vm.boot_pages vm.boot_pages: 1024 ...keith ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: UMA: Increase vm.boot_pages on Dell R920 r279210
On Mar 24, 2015, at 04:19, kwh...@site.uottawa.ca wrote: I'm using /boot/loader.conf. Is there another place I should be doing this? No, that's correct, but apparently there's a problem: the RDTUN sysctl is not picked up early enough. Can you try this patch? I haven't really tested it. :-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 79665ba..a764788 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -134,8 +134,9 @@ long first_page; int vm_page_zero_count; static int boot_pages = UMA_BOOT_PAGES; -SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN, boot_pages, 0, - number of pages allocated for bootstrapping the VM system); +SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, +boot_pages, 0, +number of pages allocated for bootstrapping the VM system); static int pa_tryrelock_restart; SYSCTL_INT(_vm, OID_AUTO, tryrelock_restart, CTLFLAG_RD, @@ -349,6 +350,7 @@ vm_page_startup(vm_offset_t vaddr) * Allocate memory for use when boot strapping the kernel memory * allocator. */ + TUNABLE_INT_FETCH(vm.boot_pages, boot_pages); new_end = end - (boot_pages * UMA_SLAB_SIZE); new_end = trunc_page(new_end); mapped = pmap_map(vaddr, new_end, end, @@ -443,7 +445,7 @@ vm_page_startup(vm_offset_t vaddr) -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: UMA: Increase vm.boot_pages on Dell R920 r279210
On Tue, 24 Mar 2015, Rui Paulo wrote: On Mar 24, 2015, at 04:19, kwh...@site.uottawa.ca wrote: I'm using /boot/loader.conf. Is there another place I should be doing this? No, that's correct, but apparently there's a problem: the RDTUN sysctl is not picked up early enough. Can you try this patch? I haven't really tested it. :-) diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 79665ba..a764788 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -134,8 +134,9 @@ long first_page; int vm_page_zero_count; static int boot_pages = UMA_BOOT_PAGES; -SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN, boot_pages, 0, - number of pages allocated for bootstrapping the VM system); +SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, +boot_pages, 0, +number of pages allocated for bootstrapping the VM system); static int pa_tryrelock_restart; SYSCTL_INT(_vm, OID_AUTO, tryrelock_restart, CTLFLAG_RD, @@ -349,6 +350,7 @@ vm_page_startup(vm_offset_t vaddr) * Allocate memory for use when boot strapping the kernel memory * allocator. */ + TUNABLE_INT_FETCH(vm.boot_pages, boot_pages); new_end = end - (boot_pages * UMA_SLAB_SIZE); new_end = trunc_page(new_end); mapped = pmap_map(vaddr, new_end, end, @@ -443,7 +445,7 @@ vm_page_startup(vm_offset_t vaddr) -- Rui Paulo Patch tried. Success! I now get this after setting vm.boot_pages=1024 in /boot/loader.conf: Booting... GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #1: Tue Mar 24 13:44:48 UTC 2015 root@:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.5.1 (tags/RELEASE_351/final 225668) 20150115 WARNING: WITNESS option enabled, expect reduced performance. UMA startup boot_pages: 1024 ... And can start all 120 processors. Thanks! ...keith -- Keith White, genie.uottawa.ca engineering.uottawa.ca kwh...@uottawa.ca [+1 613 562 5800 x6681] ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
