Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/18/12 6:13 PM, Andre Oppermann wrote: On 18.11.2012 15:05, Andrey Zonov wrote: On 11/11/12 3:04 AM, Andre Oppermann wrote: On 10.11.2012 23:24, Alfred Perlstein wrote: On 11/10/12 11:18 AM, Andre Oppermann wrote: On 10.11.2012 19:04, Peter Wemm wrote: This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. Yes, that would be optimal. No it would not. I used to be able to tell people hey just try increasing maxusers and they would and suddenly the box would be OK. Now I'll have to remember 3,4,5,10,20x tunable to increase? No. The whole mbuf and cluster stuff isn't allocated or reserved at boot time. We simply need a limit to prevent it from exhausting all available kvm / physical memory whichever is less. For now, we have limit which does not allow to run even one igb(4) NIC in 9k jumbo configuration. My patch for mbuf* zone auto-sizing does fix that, or not? Are you talking about r242910? I have not tried it. -- Andrey Zonov signature.asc Description: OpenPGP digital signature
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/11/12 3:04 AM, Andre Oppermann wrote: On 10.11.2012 23:24, Alfred Perlstein wrote: On 11/10/12 11:18 AM, Andre Oppermann wrote: On 10.11.2012 19:04, Peter Wemm wrote: This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. Yes, that would be optimal. No it would not. I used to be able to tell people hey just try increasing maxusers and they would and suddenly the box would be OK. Now I'll have to remember 3,4,5,10,20x tunable to increase? No. The whole mbuf and cluster stuff isn't allocated or reserved at boot time. We simply need a limit to prevent it from exhausting all available kvm / physical memory whichever is less. For now, we have limit which does not allow to run even one igb(4) NIC in 9k jumbo configuration. Other than that there is no relation to maxusers except historic behavior. So the ideal mbuf limit is just short of keeling the kernel over no matter what maxusers says. There also isn't much to tune then as the only fix would be to add more physical ram. -- Andrey Zonov signature.asc Description: OpenPGP digital signature
Re: svn commit: r242847 - in head/sys: i386/include kern
On 18.11.2012 15:05, Andrey Zonov wrote: On 11/11/12 3:04 AM, Andre Oppermann wrote: On 10.11.2012 23:24, Alfred Perlstein wrote: On 11/10/12 11:18 AM, Andre Oppermann wrote: On 10.11.2012 19:04, Peter Wemm wrote: This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. Yes, that would be optimal. No it would not. I used to be able to tell people hey just try increasing maxusers and they would and suddenly the box would be OK. Now I'll have to remember 3,4,5,10,20x tunable to increase? No. The whole mbuf and cluster stuff isn't allocated or reserved at boot time. We simply need a limit to prevent it from exhausting all available kvm / physical memory whichever is less. For now, we have limit which does not allow to run even one igb(4) NIC in 9k jumbo configuration. My patch for mbuf* zone auto-sizing does fix that, or not? -- Andre ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/10/12 11:33 PM, Alexey Dokuchaev wrote: On Sat, Nov 10, 2012 at 10:22:52PM -0800, Peter Wemm wrote: We've had kern.ipc.nmbclusters for years. It is simple to understand, easy to predict the outcome of a change, is runtime adjustable, is a *cap* and not a reservation (like it used to be) and does not require a reboot like maxusers tweaking would, and doesn't change maxproc as a side effect. Tune the caps sensibly by default and move on. maxusers must die. Big +1. I would also like to get rid of maxusers altogether. ./danfe You aren't really understanding it. The contrived examples Peter are giving are *wrong*. Q: How do I get an extra 2MB of clusters? A1: increase kern.ipc.nmbclusters by 1000 A2: well, 2MB is 1000 clusters, and since you have mode than 768M of ram, you divide that 1000 clusters by 16 to get 62, then multiply that by 8 to reverse the maxusers slope factor above 768M, so you need to find the maxusers value you have and add 496 to it. That will probably increase your clusters by 2MB. Maybe. I am 100% aware that nmbclusters can override maxusers values, in fact it does. The real conversation goes like this: user: Why is my box seeing terrible network performance? bsdguy: Increase nmbclusters. user: what is that? bsdguy: Oh those are the mbufs, just tell me your current value. user: oh it's like 128000 bsdguy: hmm try doubling that, go sysctl kern.ipc.nmbclusters=512000 on the command line. user: ok an hour passes ... user: hmm now I can't fork any more copies of apache.. bsdguy: oh, ok, you need to increase maxproc for that. user: so sysctl kern.ipc.maxproc=1? bsdguy: no... one second... bsdguy: ok, so that's sysctl kern.maxproc=1 user: ok... bbiaf user: so now i'm getting log messages about can't open sockets... bsdguy: oh you need to increase sockets bro... one second... user: sysctl kern.maxsockets? bsdguy: oh no.. it's actually back to kern.ipc.maxsockets user: alrighty then.. bsdguy: so how is freebsd since I helped you tune it? user: well i kept hitting other resource limits, boss made me switch to Linux, it works out of the box and doesn't require an expert tuner to run a large scale server. Y'know as a last ditch effort I looked around for this 'maxusers' thing but it seems like some eggheads retired it and instead of putting my job at risk, I just went with Linux, no one gets fired for using Linux. bsdguy: managers are lame! user: yeah! managers... -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sun, Nov 11, 2012 at 1:41 AM, Albert Perlstein bri...@mu.org wrote: The real conversation goes like this: user: Why is my box seeing terrible network performance? bsdguy: Increase nmbclusters. user: what is that? bsdguy: Oh those are the mbufs, just tell me your current value. user: oh it's like 128000 bsdguy: hmm try doubling that, go sysctl kern.ipc.nmbclusters=512000 on the command line. user: ok an hour passes ... user: hmm now I can't fork any more copies of apache.. bsdguy: oh, ok, you need to increase maxproc for that. user: so sysctl kern.ipc.maxproc=1? bsdguy: no... one second... bsdguy: ok, so that's sysctl kern.maxproc=1 user: ok... bbiaf user: so now i'm getting log messages about can't open sockets... bsdguy: oh you need to increase sockets bro... one second... user: sysctl kern.maxsockets? bsdguy: oh no.. it's actually back to kern.ipc.maxsockets user: alrighty then.. bsdguy: so how is freebsd since I helped you tune it? user: well i kept hitting other resource limits, boss made me switch to Linux, it works out of the box and doesn't require an expert tuner to run a large scale server. Y'know as a last ditch effort I looked around for this 'maxusers' thing but it seems like some eggheads retired it and instead of putting my job at risk, I just went with Linux, no one gets fired for using Linux. bsdguy: managers are lame! user: yeah! managers... -Alfred Now Albert.. I know that deliberately playing dumb is fun, but there is no network difference between doubling kern.maxusers in loader.conf (the only place it can be set, it isn't runtime tuneable) and doubling kern.ipc.nmbclusters in the same place. We've always allowed people to fine-tune derived settings at runtime where it is possible. My position still is that instead of trying to dick around with maxusers curve slopes to try and somehow get the scaling right, we should instead be setting sensibly right from the start, by default. The current scaling was written when we had severe kva constraints, did reservations, etc. Now they're a cap on dynamic allocators on most platforms. Sensible defaults would be *way* higher than the current maxusers derived scaling curves. My quick survey: 8G ram - 65088 clusters - clusters capped at 6.2% of physical ram (running head) 3.5G ram - 25600 clusters - clusters capped at 5.0% of physical ram (running an old head) 32G ram - 25600 clusters - clusters capped at 1.5% of physical ram (running 9.1-stable) 72G ram - 25600 clusters - clusters capped at 0.06% of physical ram (9.1-stable again) As I've been saying from the beginning.. As these are limits on dynamic allocators, not reservations, they should be as high as we can comfortably set them without risking running out of other resources. As the code stands now.. the derived limits for 4k, 9k and 16k jumbo clusters is approximately the same space as 2K clusters. (ie: 1 x 4k cluster per 2 x 2k clusters, 1 x 16k cluster per 8 2k clusters, and so on). If we set a constant 6% for nmbclusters (since that's roughly where we're at now for smaller machines after albert's changes), then the worse case scenarios for 4k, 9k and 16k clusters are 6% each. ie: 24% of wired, physical ram. Plus all the other values derived from the nmbclusters tunable at boot. I started writing this with the intention of suggesting 10% but that might be a bit high given that: kern_mbuf.c:nmbjumbop = nmbclusters / 2; kern_mbuf.c:nmbjumbo9 = nmbclusters / 4; kern_mbuf.c:nmbjumbo16 = nmbclusters / 8; .. basically quadruples the worst case limits. Out of the box, 6% is infinitely better than we 0.06% we currently get on a 9-stable machine with 72G ram. But I object to dicking around with maxusers to derive network buffer space default limits. If we settle on something like 6%, then it should be 6%. That's easy to document and explain the meaning of the tunable. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
Alfred, You're thinking about it one step ahead, not 5 steps ahead. One step ahead: let's fix maxuser scaling. 5 steps ahead: Let's find all of the non-dynamic things that maxusers scales, figure out how to make them run-time tunable, and then make a maxusers.sh user-land script that scales these values when you type 'maxusers 512'. Then you can fiddle in userland in your hearts content with how to scale these things. The only recent time I've seen the need for ridiculously large non-default values for mbuf clusters is for one of the 10ge NICs that preallocates them at device startup time, and fails to attach right if they're not all available. With everything dynamically tunable and the maxusers script in userland, you can: * fondle the curves as you want; * export usage stats for all the things that the above tuning does affect (which you've been doing with netstat and mbuf allocation hangs, thankyou!) * start looking at providing much better inspection and auto-tuning tools, which allow the sysadmin to actually start controlling what spirally death their server decides to visit, versus just spirally death. Adrian ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
I think there are two issue here. One: you have much better idea of how to tune nmbclusters than I do. Cool! Please put that into the code. I really think that's great and the time you've pit into giving it serious thought is helpful to all. Two: you want to divorce nmbclusters (and therefor maxsockets and some other tunables) from maxusers even though that has been the way to flip a big switch for ages now. This is think is very wrong. oh you only have to change 1 thing! Wait... What was that sound? Oh it was the flushing of a toilet that was flushing down 15 years of mailing list information, FAQs and user knowledge down the toilet because the word maxusers is no longer hip to the community. That is bad. Please don't do that. On Nov 11, 2012, at 2:53 AM, Peter Wemm pe...@wemm.org wrote: On Sun, Nov 11, 2012 at 1:41 AM, Albert Perlstein bri...@mu.org wrote: The real conversation goes like this: user: Why is my box seeing terrible network performance? bsdguy: Increase nmbclusters. user: what is that? bsdguy: Oh those are the mbufs, just tell me your current value. user: oh it's like 128000 bsdguy: hmm try doubling that, go sysctl kern.ipc.nmbclusters=512000 on the command line. user: ok an hour passes ... user: hmm now I can't fork any more copies of apache.. bsdguy: oh, ok, you need to increase maxproc for that. user: so sysctl kern.ipc.maxproc=1? bsdguy: no... one second... bsdguy: ok, so that's sysctl kern.maxproc=1 user: ok... bbiaf user: so now i'm getting log messages about can't open sockets... bsdguy: oh you need to increase sockets bro... one second... user: sysctl kern.maxsockets? bsdguy: oh no.. it's actually back to kern.ipc.maxsockets user: alrighty then.. bsdguy: so how is freebsd since I helped you tune it? user: well i kept hitting other resource limits, boss made me switch to Linux, it works out of the box and doesn't require an expert tuner to run a large scale server. Y'know as a last ditch effort I looked around for this 'maxusers' thing but it seems like some eggheads retired it and instead of putting my job at risk, I just went with Linux, no one gets fired for using Linux. bsdguy: managers are lame! user: yeah! managers... -Alfred Now Albert.. I know that deliberately playing dumb is fun, but there is no network difference between doubling kern.maxusers in loader.conf (the only place it can be set, it isn't runtime tuneable) and doubling kern.ipc.nmbclusters in the same place. We've always allowed people to fine-tune derived settings at runtime where it is possible. My position still is that instead of trying to dick around with maxusers curve slopes to try and somehow get the scaling right, we should instead be setting sensibly right from the start, by default. The current scaling was written when we had severe kva constraints, did reservations, etc. Now they're a cap on dynamic allocators on most platforms. Sensible defaults would be *way* higher than the current maxusers derived scaling curves. My quick survey: 8G ram - 65088 clusters - clusters capped at 6.2% of physical ram (running head) 3.5G ram - 25600 clusters - clusters capped at 5.0% of physical ram (running an old head) 32G ram - 25600 clusters - clusters capped at 1.5% of physical ram (running 9.1-stable) 72G ram - 25600 clusters - clusters capped at 0.06% of physical ram (9.1-stable again) As I've been saying from the beginning.. As these are limits on dynamic allocators, not reservations, they should be as high as we can comfortably set them without risking running out of other resources. As the code stands now.. the derived limits for 4k, 9k and 16k jumbo clusters is approximately the same space as 2K clusters. (ie: 1 x 4k cluster per 2 x 2k clusters, 1 x 16k cluster per 8 2k clusters, and so on). If we set a constant 6% for nmbclusters (since that's roughly where we're at now for smaller machines after albert's changes), then the worse case scenarios for 4k, 9k and 16k clusters are 6% each. ie: 24% of wired, physical ram. Plus all the other values derived from the nmbclusters tunable at boot. I started writing this with the intention of suggesting 10% but that might be a bit high given that: kern_mbuf.c:nmbjumbop = nmbclusters / 2; kern_mbuf.c:nmbjumbo9 = nmbclusters / 4; kern_mbuf.c:nmbjumbo16 = nmbclusters / 8; .. basically quadruples the worst case limits. Out of the box, 6% is infinitely better than we 0.06% we currently get on a 9-stable machine with 72G ram. But I object to dicking around with maxusers to derive network buffer space default limits. If we settle on something like 6%, then it should be 6%. That's easy to document and explain the meaning of the tunable. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If
Re: svn commit: r242847 - in head/sys: i386/include kern
Oh, OK, I didn't know it was so involved. I probably don't have anything to worry about then. :) -Alfred On 11/11/12 8:52 AM, Adrian Chadd wrote: Alfred, You're thinking about it one step ahead, not 5 steps ahead. One step ahead: let's fix maxuser scaling. 5 steps ahead: Let's find all of the non-dynamic things that maxusers scales, figure out how to make them run-time tunable, and then make a maxusers.sh user-land script that scales these values when you type 'maxusers 512'. Then you can fiddle in userland in your hearts content with how to scale these things. The only recent time I've seen the need for ridiculously large non-default values for mbuf clusters is for one of the 10ge NICs that preallocates them at device startup time, and fails to attach right if they're not all available. With everything dynamically tunable and the maxusers script in userland, you can: * fondle the curves as you want; * export usage stats for all the things that the above tuning does affect (which you've been doing with netstat and mbuf allocation hangs, thankyou!) * start looking at providing much better inspection and auto-tuning tools, which allow the sysadmin to actually start controlling what spirally death their server decides to visit, versus just spirally death. Adrian ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11 November 2012 09:11, Alfred Perlstein bri...@mu.org wrote: Oh, OK, I didn't know it was so involved. I probably don't have anything to worry about then. :) Nono - I want you to worry about it. But _I_ want there to be a slightly longer term goal that's less about dictating policy in the kernel and more about providing both the flexibility _and_ the feedback so you and others can do this kind of thing in userspace without needing to hack the kernel or recompile. So Alfred - are you up to the challenge? :) Enumerate _every_ thing that maxusers tunes at startup time and make sure it's tunable at run-time? Once that's done, we can turn maxusers into a userland _command_ that can be run _anytime_, and you all can bikeshed over the tuning of _that_ thing until we die from heat death. That way everyone wins. :) adrian ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/11/12 12:31 PM, Adrian Chadd wrote: On 11 November 2012 09:11, Alfred Perlstein bri...@mu.org wrote: Oh, OK, I didn't know it was so involved. I probably don't have anything to worry about then. :) Nono - I want you to worry about it. But _I_ want there to be a slightly longer term goal that's less about dictating policy in the kernel and more about providing both the flexibility _and_ the feedback so you and others can do this kind of thing in userspace without needing to hack the kernel or recompile. So Alfred - are you up to the challenge? :) Enumerate _every_ thing that maxusers tunes at startup time and make sure it's tunable at run-time? Once that's done, we can turn maxusers into a userland _command_ that can be run _anytime_, and you all can bikeshed over the tuning of _that_ thing until we die from heat death. That way everyone wins. :) adrian Not really, I'm kind of stuck figuring out the following things: 1) why if_lagg doesn't get full bandwidth when in point to point mode. 2) why nfs has regressed with forced unmounts. 3) other critical performance issues wrt nfs,zfs,io. 4) ... other stuff more important. I can't devote any further effort to this maxusers thing, as I see it, I've submitted a fix that is a big step forward for all. If people want to run with this and get excited about it, I am glad. Unfortunately just because I found a light switch and turned it on, doesn't suddenly mean that I owe the community a fancy new vanity installed. After all, it is just a toilet, now that I don't have to worry about everyone pissing all over the seat, my job is done. There are more important goals. I'm not playing the you touched it, it's your responsibility game any longer. this is why no one has bothered to fix it, because the community feels that it is ok to saddle someone making a small incremental change with an impossibly hard halting problem of nonsense. So, yeah feel free to install that vanity, but don't make me sorry I flipped a switch. This is why we all carry flashlights around (our personal sysctl.conf), because we're afraid of this community reaction and wonder why there's piss on the seat. I think Bruce said it best: But don't ask alfred to fix all the old mistakes. This will be my last email on this issue. -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
on 10/11/2012 04:56 Alfred Perlstein said the following: Sure, this is magic for i386 PAE machines. 384 maxusers was pretty much the highest you wanted auto-tuned SAFELY for 32bit KVA on i386. So 384 is i386 sans 'i' and minus two? :-) Sorry, I still couldn't find an explanation of 384 in your reply... E.g. why not 320 or 400. -- Andriy Gapon ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/10/12 2:50 AM, Andriy Gapon wrote: on 10/11/2012 04:56 Alfred Perlstein said the following: Sure, this is magic for i386 PAE machines. 384 maxusers was pretty much the highest you wanted auto-tuned SAFELY for 32bit KVA on i386. So 384 is i386 sans 'i' and minus two? :-) Sorry, I still couldn't find an explanation of 384 in your reply... E.g. why not 320 or 400. Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. -- Eitan Adler Source, Ports, Doc committer Bugmeister, Ports Security teams ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/10/12 8:25 AM, Eitan Adler wrote: On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. I will paste the relevant log messages which do explain it. If you would like to add a comment or work on a comment that won't be criticized for being too wordy then we can do that together. r89769 | dillon | 2002-01-24 17:54:16 -0800 (Thu, 24 Jan 2002) | 9 lines Make the 'maxusers 0' auto-sizing code slightly more conservative. Change from 1 megabyte of ram per user to 2 megabytes of ram per user, and reduce the cap from 512 to 384. 512 leaves around 240 MB of KVM available while 384 leaves 270 MB of KVM available. Available KVM is important in order to deal with zalloc and kernel malloc area growth. Reviewed by:mckusick MFC: either before 4.5 if re's agree, or after 4.5 r87546 | dillon | 2001-12-08 17:57:09 -0800 (Sat, 08 Dec 2001) | 6 lines Allow maxusers to be specified as 0 in the kernel config, which will cause the system to auto-size to between 32 and 512 depending on the amount of memory. MFC after: 1 week ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/10/12 8:38 AM, Alfred Perlstein wrote: On 11/10/12 8:25 AM, Eitan Adler wrote: On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. I will paste the relevant log messages which do explain it. If you would like to add a comment or work on a comment that won't be criticized for being too wordy then we can do that together. r89769 | dillon | 2002-01-24 17:54:16 -0800 (Thu, 24 Jan 2002) | 9 lines Make the 'maxusers 0' auto-sizing code slightly more conservative. Change from 1 megabyte of ram per user to 2 megabytes of ram per user, and reduce the cap from 512 to 384. 512 leaves around 240 MB of KVM available while 384 leaves 270 MB of KVM available. Available KVM is important in order to deal with zalloc and kernel malloc area growth. Reviewed by:mckusick MFC: either before 4.5 if re's agree, or after 4.5 r87546 | dillon | 2001-12-08 17:57:09 -0800 (Sat, 08 Dec 2001) | 6 lines Allow maxusers to be specified as 0 in the kernel config, which will cause the system to auto-size to between 32 and 512 depending on the amount of memory. MFC after: 1 week Let me add a bit of commentary on these logs... Effectively what happens is that as maxusers goes up, the amount of kernel memory needed for the structures grows. Unfortunately at a certain point the slope of this function is too steep and makes it too easy to exhaust kernel address space (which kills the crab (and the kernel)) what was decided at the time was just to cap it hard. What was really needed was for the slope to be relaxed and for architectures to be allowed to override/cap it if needed. This is now done. Hopefully it's done correctly and will last us some time. -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 10 November 2012 11:44, Alfred Perlstein bri...@mu.org wrote: On 11/10/12 8:38 AM, Alfred Perlstein wrote: On 11/10/12 8:25 AM, Eitan Adler wrote: On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. I will paste the relevant log messages which do explain it. If you would like to add a comment or work on a comment that won't be criticized for being too wordy then we can do that together. r89769 | dillon | 2002-01-24 17:54:16 -0800 (Thu, 24 Jan 2002) | 9 lines Make the 'maxusers 0' auto-sizing code slightly more conservative. Change from 1 megabyte of ram per user to 2 megabytes of ram per user, and reduce the cap from 512 to 384. 512 leaves around 240 MB of KVM available while 384 leaves 270 MB of KVM available. Available KVM is important in order to deal with zalloc and kernel malloc area growth. Reviewed by:mckusick MFC: either before 4.5 if re's agree, or after 4.5 r87546 | dillon | 2001-12-08 17:57:09 -0800 (Sat, 08 Dec 2001) | 6 lines Allow maxusers to be specified as 0 in the kernel config, which will cause the system to auto-size to between 32 and 512 depending on the amount of memory. MFC after: 1 week Let me add a bit of commentary on these logs... Effectively what happens is that as maxusers goes up, the amount of kernel memory needed for the structures grows. Unfortunately at a certain point the slope of this function is too steep and makes it too easy to exhaust kernel address space (which kills the crab (and the kernel)) what was decided at the time was just to cap it hard. I understand this. What I was confused by is where the number 384 comes from? Was this empirically tested? Mathematically deduced? Arbitrarily chosen? I think the log messages you pasted mostly answered this. -- Eitan Adler Source, Ports, Doc committer Bugmeister, Ports Security teams ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/10/12 8:48 AM, Eitan Adler wrote: On 10 November 2012 11:44, Alfred Perlstein bri...@mu.org wrote: On 11/10/12 8:38 AM, Alfred Perlstein wrote: On 11/10/12 8:25 AM, Eitan Adler wrote: On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. I will paste the relevant log messages which do explain it. If you would like to add a comment or work on a comment that won't be criticized for being too wordy then we can do that together. r89769 | dillon | 2002-01-24 17:54:16 -0800 (Thu, 24 Jan 2002) | 9 lines Make the 'maxusers 0' auto-sizing code slightly more conservative. Change from 1 megabyte of ram per user to 2 megabytes of ram per user, and reduce the cap from 512 to 384. 512 leaves around 240 MB of KVM available while 384 leaves 270 MB of KVM available. Available KVM is important in order to deal with zalloc and kernel malloc area growth. Reviewed by:mckusick MFC: either before 4.5 if re's agree, or after 4.5 r87546 | dillon | 2001-12-08 17:57:09 -0800 (Sat, 08 Dec 2001) | 6 lines Allow maxusers to be specified as 0 in the kernel config, which will cause the system to auto-size to between 32 and 512 depending on the amount of memory. MFC after: 1 week Let me add a bit of commentary on these logs... Effectively what happens is that as maxusers goes up, the amount of kernel memory needed for the structures grows. Unfortunately at a certain point the slope of this function is too steep and makes it too easy to exhaust kernel address space (which kills the crab (and the kernel)) what was decided at the time was just to cap it hard. I understand this. What I was confused by is where the number 384 comes from? Was this empirically tested? Mathematically deduced? Arbitrarily chosen? I think the log messages you pasted mostly answered this. Sure, if you'd like you can help me craft that comment now? -alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, 2012-11-10 at 08:38 -0800, Alfred Perlstein wrote: On 11/10/12 8:25 AM, Eitan Adler wrote: On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. The fact that such a thing could happen explains much about the current state of the code. An outsider could easily come to the conclusion that the FreeBSD motto is something along the lines of It should be as hard to read as it was to write. -- Ian ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === -- Eitan Adler Source, Ports, Doc committer Bugmeister, Ports Security teams ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 9:24 AM, Ian Lepore free...@damnhippie.dyndns.org wrote: On Sat, 2012-11-10 at 08:38 -0800, Alfred Perlstein wrote: On 11/10/12 8:25 AM, Eitan Adler wrote: On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. The fact that such a thing could happen explains much about the current state of the code. An outsider could easily come to the conclusion that the FreeBSD motto is something along the lines of It should be as hard to read as it was to write. Don't forget to explain that you get 1 maxusers per 2MB of physical memory which turns into 64 x 2k clusters and a whole series of side effects. Wouldn't it be nice if we could write By default, mbuf clusters are capped at 6% of physical ram or 6% of kernel address space, whichever is smaller ? That's a little easier than trying to explain maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE) if (maxusers 384) maxusers = 384 + ((maxusers - 384) / 8); nmbclusters = 1024 + maxusers * 64; I'd sure prefer to explain: /* pick smaller of kva pages or physical pages */ if ((physpages / 16) (kvapages / 16)) nmbclusters = physpages / 16; else nmbclusters = kvapages / 16; Leave maxusers for calculating maxproc. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === That's actually completely 100% incorrect... -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 10 November 2012 12:45, Peter Wemm pe...@wemm.org wrote: On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === That's actually completely 100% incorrect... okay. I'm going by the log messages posted so far. I have no idea how this works. Can you explain it better? -- Eitan Adler Source, Ports, Doc committer Bugmeister, Ports Security teams ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 9:43 AM, Peter Wemm pe...@wemm.org wrote: /* pick smaller of kva pages or physical pages */ if ((physpages / 16) (kvapages / 16)) nmbclusters = physpages / 16; else nmbclusters = kvapages / 16; (note: not actual numbers.. kva pages doesn't exist and there's a pages vs cluster size factor missing. The example was for the discussion) -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 9:45 AM, Peter Wemm pe...@wemm.org wrote: On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === That's actually completely 100% incorrect... Would it be a good idea to reference the other commit ref. numbers done by dillon@ in the commit message, or would this be redundant? Thanks! -Garrett ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 9:48 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:45, Peter Wemm pe...@wemm.org wrote: On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === That's actually completely 100% incorrect... okay. I'm going by the log messages posted so far. I have no idea how this works. Can you explain it better? That's exactly my point.. You get 1 maxuser per 2MB of physical ram. If you get more than 384 maxusers (ie: 192GB of ram) we scale it differently for the part past 192GB. I have no idea how the hell to calculate that. You get an unlimited number of regular mbufs. You get 64 clusters per maxuser (128k) Unless I fubared the numbers, this currently works out to be 6%, or 1/16. Each MD backend gets to provide a cap for maxusers, which is in units of 2MB. For an i386 PAE machine you have a finite amount of KVA space (1GB, but this is adjustable.. you can easily configure it for 3GB kva with one compile option for the kernel). The backends where the nmbclusters comes out of KVA should calculate the number of 2MB units to avoid running out of KVA. amd64 does a mixture of direct map and kva allocations. eg: mbufs and clusters come from direct map, the jumbo clusters come from kva. So side effects of nmbclusters for amd64 are more complicated. 1/2 of the nmbclusters (which are in physcal ram) are allocated as jumbo frames (kva) 1/4 of nmbclusters (physical) are 9k jumbo frames (kva) 1/8 of nmbclusters (physical) are used to set the 16k kva backed jumbo frame pool. amd64 kva is large enough now, but my recollection is that sparc64 has a small kva plus a large direct map. Tuning for amd64 isn't relevant for sparc64. mips has direct map, but doesn't have a large direct map, nor a large kva. This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 9:51 AM, Garrett Cooper yaneg...@gmail.com wrote: On Sat, Nov 10, 2012 at 9:45 AM, Peter Wemm pe...@wemm.org wrote: On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === That's actually completely 100% incorrect... Would it be a good idea to reference the other commit ref. numbers done by dillon@ in the commit message, or would this be redundant? Sadly not. There's so many layers of indirection and obfuscation of units that there isn't a commit message that explains the current state. You have to read the code and reverse the math. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, 10 Nov 2012, Eitan Adler wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === The code and the original log message are much clearer. Saying 2MB total in the above significantly obfuscates the 2MB in the code. As documented in the log message, it is a heuristic scale factor for scaling from the _physical_ memory size to the maxusers parameter. It is also clear in the code that it doesn't scale from a virtual memory size, but less clear that it scales to something related to a virtual memory size (it scales to maxusers which is used to tune parametrize many things including kva, and the 384 is mainly to limit it for the kva things. This is bogus of course). The 2MB is certainly not for kva, and even more certainly, the total kva is not 2MB. slope of the function is an unclear way of describing the scaling steps. Originally, there was only the scaling step involving scaling physical memory by 2MB. Scaling was stopped when maxusers hit 384: maxusers was clamped at 384. This corresponds to changing the slope of the function to 0. Now we change it to 1/8 of the 1/2MB scale, i.e., to incrementing maxusers by 1 for every 16MB of _physical_ memory. This is still bogus, but probably better than clamping. Describing this is complicated by the new VM_MAX_AUTOTUNE_MAX_USERS parameter. This gives a clamp on maxusers corresponding to the old one of 384 when VM_MAX_AUTOTUNE_MAX_USERS is defined to be that value. This is needed mainly for PAE. I don't like this value being ifdefed. For PAE, the value just can't be increased (except as described below), and decreasing it is not useful. Other unclarities in the above: - its first sentence says that the limit is to a fixed cap, but the whole point of recent changes is to change the limit from a fixed cap to a variable one. The second sentence describes the old behviour with the fixed cap. The third sentence modifies the first 2. Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. Original log message: % % revision 1.51 % date: 2002/01/25 01:54:16; author: dillon; state: Exp; lines: +3 -3 % Make the 'maxusers 0' auto-sizing code slightly more conservative. Change % from 1 megabyte of ram per user to 2 megabytes of ram per user, and % reduce the cap from 512 to 384. 512 leaves around 240 MB of KVM available % while 384 leaves 270 MB of KVM available. Available KVM is important % in order to deal with zalloc and kernel malloc area growth. % % Reviewed by: mckusick % MFC: either before 4.5 if re's agree, or after 4.5 All these magic numbers are completely wrong if someone changes VM_MIN_KERNEL_ADDRESS, as is very reasonable if they run out of of kva. There is a normal if not working option for this (KVA_PAGES), unlike for VM_MAX_AUTOTUNE_MAX_USERS or the old hard-coded 384. Doubling of the i386 KVA_PAGES from 256 to 512 should approximately double many of the above numbers and more than double others (when the kva consumed by say the buffer cache is not doubled to match, we can have more than double the kva for mbufs. There are other bogus limits and tuning of the buffer cache based on a fixed limit and physical memory, and doubling KVA_PAGES doesn't double all of these). PAE already doubles KVA_SIZE, so all of the above numbers must be off by a factor of about 2 for either PAE or !PAE. The magic numbers are also made significantly wrong by any significant use of the buffer cache limit to increase the amount of kva for the buffer cache. Anyone who does this must know what they are doing, but isn't helped by hard-coded magic number in comments. The above numbers are probably slightly made slightly wrong by general bloat. If 384 gave 270 MB free in 201, it must give 270 MB in 2012. Bruce ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 10.11.2012 19:04, Peter Wemm wrote: On Sat, Nov 10, 2012 at 9:48 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:45, Peter Wemm pe...@wemm.org wrote: On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === That's actually completely 100% incorrect... okay. I'm going by the log messages posted so far. I have no idea how this works. Can you explain it better? That's exactly my point.. You get 1 maxuser per 2MB of physical ram. If you get more than 384 maxusers (ie: 192GB of ram) we scale it differently for the part past 192GB. I have no idea how the hell to Rather past 768MB of RAM. calculate that. You get an unlimited number of regular mbufs. You get 64 clusters per maxuser (128k) Unless I fubared the numbers, this currently works out to be 6%, or 1/16. Each MD backend gets to provide a cap for maxusers, which is in units of 2MB. For an i386 PAE machine you have a finite amount of KVA space (1GB, but this is adjustable.. you can easily configure it for 3GB kva with one compile option for the kernel). The backends where the nmbclusters comes out of KVA should calculate the number of 2MB units to avoid running out of KVA. amd64 does a mixture of direct map and kva allocations. eg: mbufs and clusters come from direct map, the jumbo clusters come from kva. So side effects of nmbclusters for amd64 are more complicated. 1/2 of the nmbclusters (which are in physcal ram) are allocated as jumbo frames (kva) 1/4 of nmbclusters (physical) are 9k jumbo frames (kva) 1/8 of nmbclusters (physical) are used to set the 16k kva backed jumbo frame pool. The mbufs and clusters of different types are not allocated at startup time, but rather their total allocation at runtime is *limited* to that maximal value in UMA. amd64 kva is large enough now, but my recollection is that sparc64 has a small kva plus a large direct map. Tuning for amd64 isn't relevant for sparc64. mips has direct map, but doesn't have a large direct map, nor a large kva. This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. Yes, that would be optimal. -- Andre ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, 10 Nov 2012, Peter Wemm wrote: On Sat, Nov 10, 2012 at 9:24 AM, Ian Lepore free...@damnhippie.dyndns.org wrote: On Sat, 2012-11-10 at 08:38 -0800, Alfred Perlstein wrote: It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. The fact that such a thing could happen explains much about the current state of the code. An outsider could easily come to the conclusion that the FreeBSD motto is something along the lines of It should be as hard to read as it was to write. Don't forget to explain that you get 1 maxusers per 2MB of physical memory which turns into 64 x 2k clusters and a whole series of side effects. Wouldn't it be nice if we could write By default, mbuf clusters are capped at 6% of physical ram or 6% of kernel address space, whichever is smaller ? That's a little easier than trying to explain maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE) if (maxusers 384) maxusers = 384 + ((maxusers - 384) / 8); nmbclusters = 1024 + maxusers * 64; I'd sure prefer to explain: /* pick smaller of kva pages or physical pages */ if ((physpages / 16) (kvapages / 16)) nmbclusters = physpages / 16; else nmbclusters = kvapages / 16; Leave maxusers for calculating maxproc. I prefer to write clear code and not echo it in comments: nmbclusters = min(physpages, kvapages) / 16; The most interesting point (the reason why 6% was chosen) is not documented by either. Unfortunately, this is too simple to work - probably min() has a wrong type. - probably physical == virtual limit is too simple - varies scaling errors for the magic 16. It assumes that the cluster size is the page size. Your comment avoids this problem by saying mbuf clusters and not giving their units, so that is correct if the units are the same as for the memory sizes. But don't ask alfred to fix all the old mistakes. To change to the above, you at least have to subtract the contribution of nmbclusters to maxusers and change related magic and check that everything fits again. Lots of separate limits risks unduly restricting each while not actually preventing over-commit unless all the limits are unduly restrictive. Bruce ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/10/12 11:18 AM, Andre Oppermann wrote: On 10.11.2012 19:04, Peter Wemm wrote: This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. Yes, that would be optimal. No it would not. I used to be able to tell people hey just try increasing maxusers and they would and suddenly the box would be OK. Now I'll have to remember 3,4,5,10,20x tunable to increase? The concept of a single knob to do ***basic*** tuning is a good one. Please leave it alone. There is nothing about maxusers that stops someone from tuning individual subsystems. You just wind up making FreeBSD an experts only playground by gratuitously changing this. Again, please leave it alone, we need basic tuning to be simple and easy. What we have now works. Do not pull it apart and make it convoluted and expert only. thank you, -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/10/12 9:24 AM, Ian Lepore wrote: On Sat, 2012-11-10 at 08:38 -0800, Alfred Perlstein wrote: On 11/10/12 8:25 AM, Eitan Adler wrote: On 10 November 2012 11:19, Alfred Perlstein bri...@mu.org wrote: Please consult the svn log for this file, it's relatively clear just in the commit logs/comments. Grep for 384/512 and look around. Can this reasoning be added as a comment? I did grep for 384 in the log, but a) I didn't find the answer b) one shouldn't have to. It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. The fact that such a thing could happen explains much about the current state of the code. An outsider could easily come to the conclusion that the FreeBSD motto is something along the lines of It should be as hard to read as it was to write. -- Ian Ian, I am very much in favor of commenting code. In fact I would like to have more story like comments that explain things in the code. Unfortunately, EVERY TIME I DO THIS I get a bikeshed on it and it soaks up a whole day if not more. FreeBSD community needs to rally behind people, not block them for making changes like these. Why do you think no one had the balls to touch maxusers in the first place? ... If you guessed not worth the bikeshed you'd be right. -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 10.11.2012 23:24, Alfred Perlstein wrote: On 11/10/12 11:18 AM, Andre Oppermann wrote: On 10.11.2012 19:04, Peter Wemm wrote: This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. Yes, that would be optimal. No it would not. I used to be able to tell people hey just try increasing maxusers and they would and suddenly the box would be OK. Now I'll have to remember 3,4,5,10,20x tunable to increase? No. The whole mbuf and cluster stuff isn't allocated or reserved at boot time. We simply need a limit to prevent it from exhausting all available kvm / physical memory whichever is less. Other than that there is no relation to maxusers except historic behavior. So the ideal mbuf limit is just short of keeling the kernel over no matter what maxusers says. There also isn't much to tune then as the only fix would be to add more physical ram. -- Andre The concept of a single knob to do ***basic*** tuning is a good one. Please leave it alone. There is nothing about maxusers that stops someone from tuning individual subsystems. You just wind up making FreeBSD an experts only playground by gratuitously changing this. Again, please leave it alone, we need basic tuning to be simple and easy. What we have now works. Do not pull it apart and make it convoluted and expert only. thank you, -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Nov 10, 2012, at 3:04 PM, Andre Oppermann an...@freebsd.org wrote: On 10.11.2012 23:24, Alfred Perlstein wrote: On 11/10/12 11:18 AM, Andre Oppermann wrote: On 10.11.2012 19:04, Peter Wemm wrote: This is complicated but we need a simple user visible view of it. It really needs to be something like nmbclusters defaults to 6% of physical ram, with machine dependent limits. The MD limits are bad enough, and using bogo-units like maxusers just makes it worse. Yes, that would be optimal. No it would not. I used to be able to tell people hey just try increasing maxusers and they would and suddenly the box would be OK. Now I'll have to remember 3,4,5,10,20x tunable to increase? No. The whole mbuf and cluster stuff isn't allocated or reserved at boot time. We simply need a limit to prevent it from exhausting all available kvm / physical memory whichever is less. Other than that there is no relation to maxusers except historic behavior. So the ideal mbuf limit is just short of keeling the kernel over no matter what maxusers says. There also isn't much to tune then as the only fix would be to add more physical ram. I think that makes sense. If you have ideas please look into it. -Alfred ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 11:18 AM, Andre Oppermann an...@freebsd.org wrote: On 10.11.2012 19:04, Peter Wemm wrote: On Sat, Nov 10, 2012 at 9:48 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:45, Peter Wemm pe...@wemm.org wrote: On Sat, Nov 10, 2012 at 9:33 AM, Eitan Adler ead...@freebsd.org wrote: On 10 November 2012 12:04, Alfred Perlstein bri...@mu.org wrote: Sure, if you'd like you can help me craft that comment now? I think this is short and clear: === Limit the amount of kernel address space used to a fixed cap. 384 is an arbitrarily chosen value that leaves 270 MB of KVA available of the 2 MB total. On systems with large amount of memory reduce the the slope of the function in order to avoiding exhausting KVA. === That's actually completely 100% incorrect... okay. I'm going by the log messages posted so far. I have no idea how this works. Can you explain it better? That's exactly my point.. You get 1 maxuser per 2MB of physical ram. If you get more than 384 maxusers (ie: 192GB of ram) we scale it differently for the part past 192GB. I have no idea how the hell to Rather past 768MB of RAM. You're right. It is 768M ram before the slope of the cap changes. maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE); This is simple math. physpages is units of pages. (2 * 1024 * 1024 / PAGE_SIZE) = 2MB, converted to pages. So maxusers = 1 per 2MB of physical ram. 20MB of ram calculates maxusers = 10. 768M ram gives maxusers = 384. Above 768M, one maxuser is allocated per 2 * 8 = 16MB of ram. 4G gives maxusers = 592. 16G gives maxusers = 1360. calculate that. You get an unlimited number of regular mbufs. You get 64 clusters per maxuser (128k) Unless I fubared the numbers, this currently works out to be 6%, or 1/16. Each MD backend gets to provide a cap for maxusers, which is in units of 2MB. For an i386 PAE machine you have a finite amount of KVA space (1GB, but this is adjustable.. you can easily configure it for 3GB kva with one compile option for the kernel). The backends where the nmbclusters comes out of KVA should calculate the number of 2MB units to avoid running out of KVA. amd64 does a mixture of direct map and kva allocations. eg: mbufs and clusters come from direct map, the jumbo clusters come from kva. I'm aware of this. Behavior of how to set a Don't die exhaustion is MD. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 11:32 AM, Bruce Evans b...@optusnet.com.au wrote: On Sat, 10 Nov 2012, Peter Wemm wrote: On Sat, Nov 10, 2012 at 9:24 AM, Ian Lepore free...@damnhippie.dyndns.org wrote: On Sat, 2012-11-10 at 08:38 -0800, Alfred Perlstein wrote: It probably could be added, but then a bunch of other people would complain about the comment being too wordy or not in English. The fact that such a thing could happen explains much about the current state of the code. An outsider could easily come to the conclusion that the FreeBSD motto is something along the lines of It should be as hard to read as it was to write. Don't forget to explain that you get 1 maxusers per 2MB of physical memory which turns into 64 x 2k clusters and a whole series of side effects. Wouldn't it be nice if we could write By default, mbuf clusters are capped at 6% of physical ram or 6% of kernel address space, whichever is smaller ? That's a little easier than trying to explain maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE) if (maxusers 384) maxusers = 384 + ((maxusers - 384) / 8); nmbclusters = 1024 + maxusers * 64; I'd sure prefer to explain: /* pick smaller of kva pages or physical pages */ if ((physpages / 16) (kvapages / 16)) nmbclusters = physpages / 16; else nmbclusters = kvapages / 16; Leave maxusers for calculating maxproc. I prefer to write clear code and not echo it in comments: nmbclusters = min(physpages, kvapages) / 16; The most interesting point (the reason why 6% was chosen) is not documented by either. The divide by 16 was based on observation of existing behavior, notwithstanding my confusion over the point that the slope change kicks in. nmbclusters = 1024 + maxusers * 64; Aside from the elevated starting point, that is 64 x 2K clusters per maxuser bogo-unit. Before the slope change, 2MB ram gives 1 maxuser bogo-unit, which gives 128K of clusters. 2M / 128K = 16, or 6.25%. After the slope change, 16MB ram gives 1 maxuser bogo-unit. 16M / 128k = 128.. or 0.78%. ie: current autoconfiguration is roughly 6% of ram below 768MB, and less than 1% of ram above 768MB. Given that the allocator has changed significantly over the last few years and is a cap now, not a fixed reservation, we should be setting these limits to a comfortable safe limit. Back when it was a fixed reservation we had far more in the way of constraints. Telling people increase maxusers 10x is silly. maxusers sets two things directly. maxproc and nmbclusers. maxproc = maxusers * 16. Having people increase maxproc as a side effect of trying to increase network buffers is not helpful. Having people do post-defaults tuning in understandable units is much more likely to give predictable results. Which is easier? Q: How do I get an extra 2MB of clusters? A1: increase kern.ipc.nmbclusters by 1000 A2: well, 2MB is 1000 clusters, and since you have mode than 768M of ram, you divide that 1000 clusters by 16 to get 62, then multiply that by 8 to reverse the maxusers slope factor above 768M, so you need to find the maxusers value you have and add 496 to it. That will probably increase your clusters by 2MB. Maybe. We've had kern.ipc.nmbclusters for years. It is simple to understand, easy to predict the outcome of a change, is runtime adjustable, is a *cap* and not a reservation (like it used to be) and does not require a reboot like maxusers tweaking would, and doesn't change maxproc as a side effect. Tune the caps sensibly by default and move on. maxusers must die. -- Peter Wemm - pe...@wemm.org; pe...@freebsd.org; pe...@yahoo-inc.com; KI6FJV All of this is for nothing if we don't go to the stars - JMS/B5 If Java had true garbage collection, most programs would delete themselves upon execution. -- Robert Sewell ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On Sat, Nov 10, 2012 at 10:22:52PM -0800, Peter Wemm wrote: We've had kern.ipc.nmbclusters for years. It is simple to understand, easy to predict the outcome of a change, is runtime adjustable, is a *cap* and not a reservation (like it used to be) and does not require a reboot like maxusers tweaking would, and doesn't change maxproc as a side effect. Tune the caps sensibly by default and move on. maxusers must die. Big +1. I would also like to get rid of maxusers altogether. ./danfe ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
svn commit: r242847 - in head/sys: i386/include kern
Author: alfred
Date: Sat Nov 10 02:08:40 2012
New Revision: 242847
URL: http://svnweb.freebsd.org/changeset/base/242847
Log:
Allow maxusers to scale on machines with large address space.
Some hooks are added to clamp down maxusers and nmbclusters for
small address space systems.
VM_MAX_AUTOTUNE_MAXUSERS - the max maxusers that will be autotuned based on
physical memory.
VM_MAX_AUTOTUNE_NMBCLUSTERS - max nmbclusters based on physical memory.
These are set to the old values on i386 to preserve the clamping that was
being done to all arches.
Another macro VM_AUTOTUNE_NMBCLUSTERS is provided to allow an override
for the calculation on a MD basis. Currently no arch defines this.
Reviewed by: peter
MFC after: 2 weeks
Modified:
head/sys/i386/include/vmparam.h
head/sys/kern/kern_mbuf.c
head/sys/kern/subr_param.c
Modified: head/sys/i386/include/vmparam.h
==
--- head/sys/i386/include/vmparam.h Sat Nov 10 02:08:19 2012
(r242846)
+++ head/sys/i386/include/vmparam.h Sat Nov 10 02:08:40 2012
(r242847)
@@ -202,4 +202,13 @@
#defineZERO_REGION_SIZE(64 * 1024) /* 64KB */
+#ifndef VM_MAX_AUTOTUNE_MAXUSERS
+#define VM_MAX_AUTOTUNE_MAXUSERS 384
+#endif
+
+#ifndef VM_MAX_AUTOTUNE_NMBCLUSTERS
+/* old maxusers max value. */
+#define VM_MAX_AUTOTUNE_NMBCLUSTERS (1024 + VM_MAX_AUTOTUNE_MAXUSERS * 64)
+#endif
+
#endif /* _MACHINE_VMPARAM_H_ */
Modified: head/sys/kern/kern_mbuf.c
==
--- head/sys/kern/kern_mbuf.c Sat Nov 10 02:08:19 2012(r242846)
+++ head/sys/kern/kern_mbuf.c Sat Nov 10 02:08:40 2012(r242847)
@@ -113,8 +113,17 @@ tunable_mbinit(void *dummy)
/* This has to be done before VM init. */
TUNABLE_INT_FETCH(kern.ipc.nmbclusters, nmbclusters);
- if (nmbclusters == 0)
+ if (nmbclusters == 0) {
+#ifdef VM_AUTOTUNE_NMBCLUSTERS
+ nmbclusters = VM_AUTOTUNE_NMBCLUSTERS;
+#else
nmbclusters = 1024 + maxusers * 64;
+#endif
+#ifdef VM_MAX_AUTOTUNE_NMBCLUSTERS
+ if (nmbclusters VM_MAX_AUTOTUNE_NMBCLUSTERS)
+ nmbclusters = VM_MAX_AUTOTUNE_NMBCLUSTERS;
+#endif
+ }
TUNABLE_INT_FETCH(kern.ipc.nmbjumbop, nmbjumbop);
if (nmbjumbop == 0)
Modified: head/sys/kern/subr_param.c
==
--- head/sys/kern/subr_param.c Sat Nov 10 02:08:19 2012(r242846)
+++ head/sys/kern/subr_param.c Sat Nov 10 02:08:40 2012(r242847)
@@ -278,17 +278,17 @@ init_param2(long physpages)
maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE);
if (maxusers 32)
maxusers = 32;
- /*
-* Clips maxusers to 384 on machines with = 4GB RAM or 32bit.
-* Scales it down 6x for large memory machines.
-*/
- if (maxusers 384) {
- if (sizeof(void *) = 4)
- maxusers = 384;
- else
- maxusers = 384 + ((maxusers - 384) / 6);
- }
- }
+#ifdef VM_MAX_AUTOTUNE_MAXUSERS
+if (maxusers VM_MAX_AUTOTUNE_MAXUSERS)
+maxusers = VM_MAX_AUTOTUNE_MAXUSERS;
+#endif
+/*
+ * Scales down the function in which maxusers grows once
+ * we hit 384.
+ */
+if (maxusers 384)
+maxusers = 384 + ((maxusers - 384) / 8);
+}
/*
* The following can be overridden after boot via sysctl. Note:
___
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 9 November 2012 21:08, Alfred Perlstein alf...@freebsd.org wrote: Modified: head/sys/kern/subr_param.c +#ifdef VM_MAX_AUTOTUNE_MAXUSERS +if (maxusers VM_MAX_AUTOTUNE_MAXUSERS) +maxusers = VM_MAX_AUTOTUNE_MAXUSERS; +#endif +/* + * Scales down the function in which maxusers grows once + * we hit 384. + */ +if (maxusers 384) +maxusers = 384 + ((maxusers - 384) / 8); +} Hey, I know you didn't introduce this magic number 384. But can you (or someone else) explain where it came from? -- Eitan Adler Source, Ports, Doc committer Bugmeister, Ports Security teams ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r242847 - in head/sys: i386/include kern
On 11/9/12 6:34 PM, Eitan Adler wrote: On 9 November 2012 21:08, Alfred Perlstein alf...@freebsd.org wrote: Modified: head/sys/kern/subr_param.c +#ifdef VM_MAX_AUTOTUNE_MAXUSERS +if (maxusers VM_MAX_AUTOTUNE_MAXUSERS) +maxusers = VM_MAX_AUTOTUNE_MAXUSERS; +#endif +/* + * Scales down the function in which maxusers grows once + * we hit 384. + */ +if (maxusers 384) +maxusers = 384 + ((maxusers - 384) / 8); +} Hey, I know you didn't introduce this magic number 384. But can you (or someone else) explain where it came from? Sure, this is magic for i386 PAE machines. 384 maxusers was pretty much the highest you wanted auto-tuned SAFELY for 32bit KVA on i386. Now with 64 bit machines doing away with this hard ceiling, I wanted to change the slope so that it doesn't grow too much as a conservative measure to test the waters. Take for example my amd64 machine with 16GB of ram. Without the scaling factor of maxusers = 384 + ((maxusers - 384) / 8) then I get 8173 maxusers. 8173 maxusers translates to 524096 nmbclusters (1024 + 8173 * 64). That is 2GB RAM for just nmbclusters, nevermind jumbo9 and jumbo16. With the scaling adjustment I get 1357 maxusers which is 87872 nmbclusters. That is 343MB. Somewhat more reasonable. I'm open to other suggestions and people coming in here to open the value up higher... however I wanted a conservative value for now to avoid too much concern. We'll see where this takes us. -Alfred With this patch I get 75104 nmbclusters, which is 75104 pages which is 293 MB of ram! ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
