Re: Trouble with recent auto-tuning changes
On 31.01.2013 23:25, Ian Lepore wrote: On Thu, 2013-01-31 at 18:13 +0100, Andre Oppermann wrote: On 28.01.2013 20:20, Alan Cox wrote: On 01/28/2013 08:22, Ian Lepore wrote: On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote: On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore i...@freebsd.org wrote: I ran into a panic while attempting to un-tar a large file on a DreamPlug (arm-based system) running -current. The source and dest of the un-tar is the root filesystem on sdcard, and I get this: panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated Just before the panic I see the tar process get hung in a nokva wait. 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h. In r245575 the init order for mbuf limits was changed from SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the results of sizing kernel memory. Unfortunately, the process of sizing kernel memory relies on the mbuf limits; in kmeminit(): vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; Since r245575, nmbclusters is zero when this line of code runs. If I manually plugin 32768 (the number tunable_mbinit() comes up with for this platform) in that line, the panic stops happening. So we've got two problems here... one is the circular dependency in calculating the mbuf limits. The other is the fact that some non-trivial amount of kernel memory we're allowing for mbufs is actually being used for other things. That is, if my system was actually using all the mbufs that tunable_mbinit() allowed for, then this panic while untarring a huge file would still have happened. All of this is factually correct. However, it's a red herring. The real problem is that arm, unlike every other architecture in the tree, does not enable auto-sizing of the kmem map based on the physical memory size. Specifically, you'll find VM_KMEM_SIZE_SCALE defined in arch/include/vmparam.h on every other architecture, just not on arm. This auto-sizing overrides the value of VM_KMEM_SIZE. Aha. I'll investigate what other architectures do with that and try to get the same thing going for arm. i386 or (32-bit) MIPS would be the most similar. Also, I would encourage you to look for other definitions that those architectures have that arm doesn't. As physical memory sizes continue to grow on arm-based systems, they may require other changes in vmparam.h and the machine-dependent param.h that were made on those other architectures year ago. Ian, The patch below should do the trick. Can you please test? Yep, that fixed the problem with untarring the large file. Here are some before/after numbers from sysctl, converted from bytes to KBytes for readability: vm.kmem_size_scale: 0 2 vm.kmem_map_free: 5740246440 vm.kmem_map_size: 6548 7176 vm.kmem_size: 12288253616 real memory = 536870912 (512 MB) avail memory = 516718592 (492 MB) Thank you for testing. Committed as r246204. If any other problems come up please report. -- Andre ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Trouble with recent auto-tuning changes
On 28.01.2013 20:20, Alan Cox wrote: On 01/28/2013 08:22, Ian Lepore wrote: On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote: On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore i...@freebsd.org wrote: I ran into a panic while attempting to un-tar a large file on a DreamPlug (arm-based system) running -current. The source and dest of the un-tar is the root filesystem on sdcard, and I get this: panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated Just before the panic I see the tar process get hung in a nokva wait. 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h. In r245575 the init order for mbuf limits was changed from SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the results of sizing kernel memory. Unfortunately, the process of sizing kernel memory relies on the mbuf limits; in kmeminit(): vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; Since r245575, nmbclusters is zero when this line of code runs. If I manually plugin 32768 (the number tunable_mbinit() comes up with for this platform) in that line, the panic stops happening. So we've got two problems here... one is the circular dependency in calculating the mbuf limits. The other is the fact that some non-trivial amount of kernel memory we're allowing for mbufs is actually being used for other things. That is, if my system was actually using all the mbufs that tunable_mbinit() allowed for, then this panic while untarring a huge file would still have happened. All of this is factually correct. However, it's a red herring. The real problem is that arm, unlike every other architecture in the tree, does not enable auto-sizing of the kmem map based on the physical memory size. Specifically, you'll find VM_KMEM_SIZE_SCALE defined in arch/include/vmparam.h on every other architecture, just not on arm. This auto-sizing overrides the value of VM_KMEM_SIZE. Aha. I'll investigate what other architectures do with that and try to get the same thing going for arm. i386 or (32-bit) MIPS would be the most similar. Also, I would encourage you to look for other definitions that those architectures have that arm doesn't. As physical memory sizes continue to grow on arm-based systems, they may require other changes in vmparam.h and the machine-dependent param.h that were made on those other architectures year ago. Ian, The patch below should do the trick. Can you please test? If you have any questions about any of the definitions, feel free to e-mail me. Alan P.S. When I get a little more free time, I intend to get in touch with Andre to address the apparent circular dependency. For now just know that unless that circular dependency is combined with a lack of kmem map auto-sizing, like arm, it's basically harmless. I'm working myself through it and will post a patch shortly that untangles a lot of the obscure VM initialization stuff and moves it into the modern SYSINIT world. -- Andre Index: arm/include/vmparam.h === --- arm/include/vmparam.h (revision 246082) +++ arm/include/vmparam.h (working copy) @@ -134,13 +134,16 @@ #endif #define VM_MAX_KERNEL_ADDRESS 0x + /* * Virtual size (bytes) for various kernel submaps. */ - #ifndef VM_KMEM_SIZE -#define VM_KMEM_SIZE(12*1024*1024) +#define VM_KMEM_SIZE (12*1024*1024) #endif +#ifndef VM_KMEM_SIZE_SCALE +#define VM_KMEM_SIZE_SCALE (2) +#endif #define MAXTSIZ(16*1024*1024) #ifndef DFLDSIZ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Trouble with recent auto-tuning changes
On Thu, 2013-01-31 at 18:13 +0100, Andre Oppermann wrote: On 28.01.2013 20:20, Alan Cox wrote: On 01/28/2013 08:22, Ian Lepore wrote: On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote: On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore i...@freebsd.org wrote: I ran into a panic while attempting to un-tar a large file on a DreamPlug (arm-based system) running -current. The source and dest of the un-tar is the root filesystem on sdcard, and I get this: panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated Just before the panic I see the tar process get hung in a nokva wait. 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h. In r245575 the init order for mbuf limits was changed from SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the results of sizing kernel memory. Unfortunately, the process of sizing kernel memory relies on the mbuf limits; in kmeminit(): vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; Since r245575, nmbclusters is zero when this line of code runs. If I manually plugin 32768 (the number tunable_mbinit() comes up with for this platform) in that line, the panic stops happening. So we've got two problems here... one is the circular dependency in calculating the mbuf limits. The other is the fact that some non-trivial amount of kernel memory we're allowing for mbufs is actually being used for other things. That is, if my system was actually using all the mbufs that tunable_mbinit() allowed for, then this panic while untarring a huge file would still have happened. All of this is factually correct. However, it's a red herring. The real problem is that arm, unlike every other architecture in the tree, does not enable auto-sizing of the kmem map based on the physical memory size. Specifically, you'll find VM_KMEM_SIZE_SCALE defined in arch/include/vmparam.h on every other architecture, just not on arm. This auto-sizing overrides the value of VM_KMEM_SIZE. Aha. I'll investigate what other architectures do with that and try to get the same thing going for arm. i386 or (32-bit) MIPS would be the most similar. Also, I would encourage you to look for other definitions that those architectures have that arm doesn't. As physical memory sizes continue to grow on arm-based systems, they may require other changes in vmparam.h and the machine-dependent param.h that were made on those other architectures year ago. Ian, The patch below should do the trick. Can you please test? Yep, that fixed the problem with untarring the large file. Here are some before/after numbers from sysctl, converted from bytes to KBytes for readability: vm.kmem_size_scale: 0 2 vm.kmem_map_free: 5740246440 vm.kmem_map_size: 6548 7176 vm.kmem_size: 12288253616 real memory = 536870912 (512 MB) avail memory = 516718592 (492 MB) -- Ian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Trouble with recent auto-tuning changes
On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote: On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore i...@freebsd.org wrote: I ran into a panic while attempting to un-tar a large file on a DreamPlug (arm-based system) running -current. The source and dest of the un-tar is the root filesystem on sdcard, and I get this: panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated Just before the panic I see the tar process get hung in a nokva wait. 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h. In r245575 the init order for mbuf limits was changed from SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the results of sizing kernel memory. Unfortunately, the process of sizing kernel memory relies on the mbuf limits; in kmeminit(): vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; Since r245575, nmbclusters is zero when this line of code runs. If I manually plugin 32768 (the number tunable_mbinit() comes up with for this platform) in that line, the panic stops happening. So we've got two problems here... one is the circular dependency in calculating the mbuf limits. The other is the fact that some non-trivial amount of kernel memory we're allowing for mbufs is actually being used for other things. That is, if my system was actually using all the mbufs that tunable_mbinit() allowed for, then this panic while untarring a huge file would still have happened. All of this is factually correct. However, it's a red herring. The real problem is that arm, unlike every other architecture in the tree, does not enable auto-sizing of the kmem map based on the physical memory size. Specifically, you'll find VM_KMEM_SIZE_SCALE defined in arch/include/vmparam.h on every other architecture, just not on arm. This auto-sizing overrides the value of VM_KMEM_SIZE. Aha. I'll investigate what other architectures do with that and try to get the same thing going for arm. -- Ian I arrive at the latter conclusion based on the fact that this panic happens even if no network interfaces (other than lo0) are configured. That is, nmbclusters == 0 is a reasonable approximation of my need for network mbufs. So something in the system needs to be taken into account when sizing kernel memory to allow for whatever it is about untarring a huge file that eats kernel memory (buffer cache?). I can easily reproduce this if you need me to gather any specific info. -- Ian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Trouble with recent auto-tuning changes
On 01/28/2013 08:22, Ian Lepore wrote: On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote: On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore i...@freebsd.org wrote: I ran into a panic while attempting to un-tar a large file on a DreamPlug (arm-based system) running -current. The source and dest of the un-tar is the root filesystem on sdcard, and I get this: panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated Just before the panic I see the tar process get hung in a nokva wait. 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h. In r245575 the init order for mbuf limits was changed from SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the results of sizing kernel memory. Unfortunately, the process of sizing kernel memory relies on the mbuf limits; in kmeminit(): vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; Since r245575, nmbclusters is zero when this line of code runs. If I manually plugin 32768 (the number tunable_mbinit() comes up with for this platform) in that line, the panic stops happening. So we've got two problems here... one is the circular dependency in calculating the mbuf limits. The other is the fact that some non-trivial amount of kernel memory we're allowing for mbufs is actually being used for other things. That is, if my system was actually using all the mbufs that tunable_mbinit() allowed for, then this panic while untarring a huge file would still have happened. All of this is factually correct. However, it's a red herring. The real problem is that arm, unlike every other architecture in the tree, does not enable auto-sizing of the kmem map based on the physical memory size. Specifically, you'll find VM_KMEM_SIZE_SCALE defined in arch/include/vmparam.h on every other architecture, just not on arm. This auto-sizing overrides the value of VM_KMEM_SIZE. Aha. I'll investigate what other architectures do with that and try to get the same thing going for arm. i386 or (32-bit) MIPS would be the most similar. Also, I would encourage you to look for other definitions that those architectures have that arm doesn't. As physical memory sizes continue to grow on arm-based systems, they may require other changes in vmparam.h and the machine-dependent param.h that were made on those other architectures year ago. If you have any questions about any of the definitions, feel free to e-mail me. Alan P.S. When I get a little more free time, I intend to get in touch with Andre to address the apparent circular dependency. For now just know that unless that circular dependency is combined with a lack of kmem map auto-sizing, like arm, it's basically harmless. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Trouble with recent auto-tuning changes
On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore i...@freebsd.org wrote: I ran into a panic while attempting to un-tar a large file on a DreamPlug (arm-based system) running -current. The source and dest of the un-tar is the root filesystem on sdcard, and I get this: panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated Just before the panic I see the tar process get hung in a nokva wait. 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h. In r245575 the init order for mbuf limits was changed from SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the results of sizing kernel memory. Unfortunately, the process of sizing kernel memory relies on the mbuf limits; in kmeminit(): vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE; Since r245575, nmbclusters is zero when this line of code runs. If I manually plugin 32768 (the number tunable_mbinit() comes up with for this platform) in that line, the panic stops happening. So we've got two problems here... one is the circular dependency in calculating the mbuf limits. The other is the fact that some non-trivial amount of kernel memory we're allowing for mbufs is actually being used for other things. That is, if my system was actually using all the mbufs that tunable_mbinit() allowed for, then this panic while untarring a huge file would still have happened. All of this is factually correct. However, it's a red herring. The real problem is that arm, unlike every other architecture in the tree, does not enable auto-sizing of the kmem map based on the physical memory size. Specifically, you'll find VM_KMEM_SIZE_SCALE defined in arch/include/vmparam.h on every other architecture, just not on arm. This auto-sizing overrides the value of VM_KMEM_SIZE. I arrive at the latter conclusion based on the fact that this panic happens even if no network interfaces (other than lo0) are configured. That is, nmbclusters == 0 is a reasonable approximation of my need for network mbufs. So something in the system needs to be taken into account when sizing kernel memory to allow for whatever it is about untarring a huge file that eats kernel memory (buffer cache?). I can easily reproduce this if you need me to gather any specific info. -- Ian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
