Re: maxusers and random system freezes
Hi, Despite the increased KVA space (2G now) and the perfect patch of the pthreads mechanism made by David, the server's lock-ups persist. Comparing this server's vmstat output with some other's which doesn't have the similar problem, I noticed that the FFS node value seems to be abnormally high - 75113K from 102400K possible. What becomes with the system if this value bumps even closer to the limit, and how it can be put into place? And more of it: which other parameters must be examined first? If I'll send the output of vmstat -z and vmstat -m will you look at it please? Thanks a lot, regards Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) On Mon, 9 Dec 2002, Dmitry Morozovsky wrote: Date: Mon, 9 Dec 2002 22:32:04 +0300 (MSK) From: Dmitry Morozovsky [EMAIL PROTECTED] To: Varshavchick Alexander [EMAIL PROTECTED] Cc: David Schultz [EMAIL PROTECTED], Terry Lambert [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: maxusers and random system freezes On Mon, 9 Dec 2002, Varshavchick Alexander wrote: VA the server went to a swap, because it occurs practically instantly, and VA this state goes for hours. The system is lacking some resources, or may be VA a bug somewhere, can you give any hints to it? Hmm, what about logging vmstat/pstat/netstat -m/sysctl vm ? Possibly to remote machine via remote syslog? Sincerely, D.Marck [DM5020, DM268-RIPE, DM3-RIPN] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- [EMAIL PROTECTED] *** To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
There seems to be archive posts already on the subject, the most informative of them is here: http://www.freebsd.org/cgi/getmsg.cgi?fetch=1093170+1102546+/usr/local/www/db/text/2001/freebsd-stable/20010923.freebsd-stable Did this issue got solved somehow? More specifically, how the size of the FFS node malloc area can be increased? Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) On Thu, 19 Dec 2002, Varshavchick Alexander wrote: Date: Thu, 19 Dec 2002 13:29:18 +0300 (MSK) From: Varshavchick Alexander [EMAIL PROTECTED] To: Dmitry Morozovsky [EMAIL PROTECTED] Cc: David Schultz [EMAIL PROTECTED], Terry Lambert [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: maxusers and random system freezes Hi, Despite the increased KVA space (2G now) and the perfect patch of the pthreads mechanism made by David, the server's lock-ups persist. Comparing this server's vmstat output with some other's which doesn't have the similar problem, I noticed that the FFS node value seems to be abnormally high - 75113K from 102400K possible. What becomes with the system if this value bumps even closer to the limit, and how it can be put into place? And more of it: which other parameters must be examined first? If I'll send the output of vmstat -z and vmstat -m will you look at it please? Thanks a lot, regards Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) On Mon, 9 Dec 2002, Dmitry Morozovsky wrote: Date: Mon, 9 Dec 2002 22:32:04 +0300 (MSK) From: Dmitry Morozovsky [EMAIL PROTECTED] To: Varshavchick Alexander [EMAIL PROTECTED] Cc: David Schultz [EMAIL PROTECTED], Terry Lambert [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: maxusers and random system freezes On Mon, 9 Dec 2002, Varshavchick Alexander wrote: VA the server went to a swap, because it occurs practically instantly, and VA this state goes for hours. The system is lacking some resources, or may be VA a bug somewhere, can you give any hints to it? Hmm, what about logging vmstat/pstat/netstat -m/sysctl vm ? Possibly to remote machine via remote syslog? Sincerely, D.Marck [DM5020, DM268-RIPE, DM3-RIPN] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- [EMAIL PROTECTED] *** To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Nate Lawson wrote: On Wed, 4 Dec 2002, Terry Lambert wrote: useful documentation; otherwise, I would have published what I wrote in Pentad Embedded Systems Journal already (example: the ^^^ I appreciate some of the info you give. But every time you reference a proper noun (person, journal, etc.), Google only gives results of you talking about it in FreeBSD list archives! See also freebsd mitre netbeui What kind of conclusion is one to draw from that? I'm a consistent speller? Pentad - Penton Sorry about that... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Hi David, Thanks, you're a genuis, didn't you know that? Your patch worked perfectly :) However, it didn't solve the random freezes problem. The server felt relieved a bit, load average value pushed down a little, so when the server is working, this change did him good. Now more about a state when the server sleeps. It behaves as though the load average has suddently fired to some immense value - the system responds to ping, but all other activity has almost halted. It's not like the server went to a swap, because it occurs practically instantly, and this state goes for hours. The system is lacking some resources, or may be a bug somewhere, can you give any hints to it? My best wishes Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) On Fri, 6 Dec 2002, David Schultz wrote: Date: Fri, 6 Dec 2002 05:47:24 -0800 From: David Schultz [EMAIL PROTECTED] To: Varshavchick Alexander [EMAIL PROTECTED] Cc: Terry Lambert [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: maxusers and random system freezes Thus spake Varshavchick Alexander [EMAIL PROTECTED]: On Fri, 6 Dec 2002, David Schultz wrote: Thus spake Varshavchick Alexander [EMAIL PROTECTED]: Well, now I made KVA space 2G, we'll see later on if it helps to get rid of the sudden system halts, but for some reason a side-effect has appeared: pthread_create function returns EAGAIN error now, so I had to recompile the software using it with linux threads to make it working. With the old kernel these pieces worked without problems. Can it be that somehow the enlarged KVA space messed up with the threads mechanism? I'm not a pthreads expert, but my best guess is that your program tried to create a thread with a stack address that was too high. Remember that with a 2 GB KVA, user processes have only 2 GB to play with instead of 3 GB, so attempting to mmap() a stack above about 2 GB would cause pthread_create() to return EAGAIN. Yes this makes sense, however this call to pthread_create didn't specify any special addresses for the new thread. The pthread_create was called with the NULL attribute which means that the system defaults were being used. Something in the system has gone wrong... I just glanced at the source in -STABLE, and it appears to be a pthreads bug. (Then again, maybe I'm missing something, since nobody seems to have noticed this before.) The default address at which new thread stacks are created is just below the main stack. This address is based on the lexical constant USRSTACK, but it should be initialized in uthread_init() based on the kern.usrstack value returned by sysctl. (The correct value is already used to map the main stack's red zone.) The result is that you need to make world and recompile any apps statically linked against pthreads after building your new kernel in order to get things to work. I don't have time to fiddle with pthreads until after Christmas, but you might see if the following patch (against -STABLE) helps when you reduce the configured KVA size without remaking pthreads. Index: uthread/uthread_init.c === RCS file: /home/ncvs/src/lib/libc_r/uthread/uthread_init.c,v retrieving revision 1.23.2.10 diff -u -r1.23.2.10 uthread_init.c --- uthread/uthread_init.c2002/10/22 14:44:03 1.23.2.10 +++ uthread/uthread_init.c2002/12/06 13:41:06 @@ -245,6 +245,8 @@ len = sizeof (int); if (sysctl(mib, 2, _usrstack, len, NULL, 0) == -1) _usrstack = (void *)USRSTACK; + _next_stack = _usrstack - PTHREAD_STACK_INITIAL - + PTHREAD_STACK_DEFAULT - (2 * PTHREAD_STACK_GUARD); /* * Create a red zone below the main stack. All other stacks are * constrained to a maximum size by the paramters passed to To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Mon, 9 Dec 2002, Varshavchick Alexander wrote: VA the server went to a swap, because it occurs practically instantly, and VA this state goes for hours. The system is lacking some resources, or may be VA a bug somewhere, can you give any hints to it? Hmm, what about logging vmstat/pstat/netstat -m/sysctl vm ? Possibly to remote machine via remote syslog? Sincerely, D.Marck [DM5020, DM268-RIPE, DM3-RIPN] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- [EMAIL PROTECTED] *** To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Thu, 5 Dec 2002, David Schultz wrote: In FreeBSD, each process has a unique 4G virtual address space associated with it. Not every virtual page in every address space has to be associated with real memory. Most pages can be pushed out to disk when there isn't enough free RAM, and unallocated parts of the virtual address space aren't backed by anything. (Referencing an unmapped page that the system doesn't know about generally causes the program or OS to crash. You've probably seen these as ``segmentation faults'' and ``page fault in kernel mode'' panics.) To simplify things, the kernel is mapped into a fixed location in every address space. The KVA parameter controls how big a chunk the kernel gets; the remainder goes to user processes. However, only the part of the KVA reservation that the kernel actually uses is wired to physical memory. For example, if you have a 1 GB KVA reservation and the kernel allocates only 20 MB of RAM, then only 20 MB of RAM is needed (plus some epsilon if you want to be picky), but in theory, the kernel could allocate and manage up to 1 GB of data. You don't lose extra physical memory for increasing KVA, but a large KVA size does constrain the virtual address space available to user processes. Thank you David for such an excellent explanation. So if sysctl reports vm.zone_kmem_pages: 5413 vm.zone_kmem_kvaspace: 218808320 vm.kvm_size: 1065353216 vm.kvm_free: 58720256 does it mean that total KVA reservation is 1065353216 bytes (1G) and almost all of it is really mapped to physical memory because only 58720256 (56M) is free, and the server is balancing on the edge of crashing with KVA going out? Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Thu, 5 Dec 2002, Terry Lambert wrote: ... Are you talking primarily about SHMMAXPGS=262144 option here? Then may be it'll be oevrall better to reduce it and make KVA space 2G, to leave more room for user address space? That's the one I was referring to, yes, but you didn't post your whole config (please do *NOT* post it; I can't spend the time on going over it line by line). All other config options are pretty much like the default ones. Tuning is a skill; it can be plotted out as a cookbook recipe, but it takes a lot of work to do that, and no one has volunteered. Basically, to write out a cookbook, you have to know where every byte of memory is going in the kernel, and what tunables impact each other, and how they are related. Once you know that, you could easily write a program to kick out a configuration file for various usages, or even modify the code to auto-tune itself (everything by KVA space, which impacts the base address that the kernel gets linked to... unless you compile the entire kernel PIC, which I do not recommend). But knowing the information is hard. I know it for 4.3 and 4.4. You're right, expecially for getting an _ideally_ tuned kernel. However in a real life, a specialist cannot have an absolute knowledge about _all_ server and other issues, so practical solutions are being looked for in items which can arise. Of cause, no one is arguing that a basic knowledge is needed and required. ... If you are having system freeses at random, and you want to fix them instead of living with them, some experimentation is going to be inevitable. I don't know enough about your installation to be able to give you a kernel config file to use that will magically fix all your current issues for you, and prevent future issues from coming up. That's going to have to be up to you. Surely, I'm just trying to reduce the experimental attempts as much as possible and to rise the chances of success for each new configuration version. ... No, the swap is very slightly used on this server, and the total swap size is 2G. It doesn't matter. The amount of swap the kernel allocates page tables for is based on the amount of physical RAM in the machine. You pay for the page tables whether you use them or not, for swap, for the kernel, and for any memory which you permit to be allocated at interrupt time, plus any allocations that occur after you are up and running, until you run out of physical RAM. This is one of those things you just have to know about how the kernel uses virtual memory, if you are going to be a skilled kernel tuner. As a rule, swap should be at least physical memory size + 64K on any system that you need to be able to get a system dump from, since it needs to dump physical RAM. If you are not worried about the machine falling over, then you can ignore that. Note that man tuning suggests 2* physical RAM for swap. PS: I am going to be out of touch (able to download, but not send email) for the next couple of days... up to a week. If you have more questions, and they can't wait, you will need to ask someone else. Thank you for all your advices, they've already helped a bit. Have luck in your trip or elsewhere. - Alex To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Thus spake Varshavchick Alexander [EMAIL PROTECTED]: Thank you David for such an excellent explanation. So if sysctl reports vm.zone_kmem_pages: 5413 vm.zone_kmem_kvaspace: 218808320 vm.kvm_size: 1065353216 vm.kvm_free: 58720256 does it mean that total KVA reservation is 1065353216 bytes (1G) and almost all of it is really mapped to physical memory because only 58720256 (56M) is free, and the server is balancing on the edge of crashing with KVA going out? Yes, 56 MB of unreserved kernel virtual memory (modulo fragmentation) is probably pushing it for a busy server. Try bumping KVA_PAGES. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Fri, 6 Dec 2002, David Schultz wrote: vm.zone_kmem_pages: 5413 vm.zone_kmem_kvaspace: 218808320 vm.kvm_size: 1065353216 vm.kvm_free: 58720256 does it mean that total KVA reservation is 1065353216 bytes (1G) and almost all of it is really mapped to physical memory because only 58720256 (56M) is free, and the server is balancing on the edge of crashing with KVA going out? Yes, 56 MB of unreserved kernel virtual memory (modulo fragmentation) is probably pushing it for a busy server. Try bumping KVA_PAGES. Well, now I made KVA space 2G, we'll see later on if it helps to get rid of the sudden system halts, but for some reason a side-effect has appeared: pthread_create function returns EAGAIN error now, so I had to recompile the software using it with linux threads to make it working. With the old kernel these pieces worked without problems. Can it be that somehow the enlarged KVA space messed up with the threads mechanism? Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Thus spake Varshavchick Alexander [EMAIL PROTECTED]: Well, now I made KVA space 2G, we'll see later on if it helps to get rid of the sudden system halts, but for some reason a side-effect has appeared: pthread_create function returns EAGAIN error now, so I had to recompile the software using it with linux threads to make it working. With the old kernel these pieces worked without problems. Can it be that somehow the enlarged KVA space messed up with the threads mechanism? I'm not a pthreads expert, but my best guess is that your program tried to create a thread with a stack address that was too high. Remember that with a 2 GB KVA, user processes have only 2 GB to play with instead of 3 GB, so attempting to mmap() a stack above about 2 GB would cause pthread_create() to return EAGAIN. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Fri, 6 Dec 2002, David Schultz wrote: Thus spake Varshavchick Alexander [EMAIL PROTECTED]: Well, now I made KVA space 2G, we'll see later on if it helps to get rid of the sudden system halts, but for some reason a side-effect has appeared: pthread_create function returns EAGAIN error now, so I had to recompile the software using it with linux threads to make it working. With the old kernel these pieces worked without problems. Can it be that somehow the enlarged KVA space messed up with the threads mechanism? I'm not a pthreads expert, but my best guess is that your program tried to create a thread with a stack address that was too high. Remember that with a 2 GB KVA, user processes have only 2 GB to play with instead of 3 GB, so attempting to mmap() a stack above about 2 GB would cause pthread_create() to return EAGAIN. Yes this makes sense, however this call to pthread_create didn't specify any special addresses for the new thread. The pthread_create was called with the NULL attribute which means that the system defaults were being used. Something in the system has gone wrong... Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Thus spake Varshavchick Alexander [EMAIL PROTECTED]: On Fri, 6 Dec 2002, David Schultz wrote: Thus spake Varshavchick Alexander [EMAIL PROTECTED]: Well, now I made KVA space 2G, we'll see later on if it helps to get rid of the sudden system halts, but for some reason a side-effect has appeared: pthread_create function returns EAGAIN error now, so I had to recompile the software using it with linux threads to make it working. With the old kernel these pieces worked without problems. Can it be that somehow the enlarged KVA space messed up with the threads mechanism? I'm not a pthreads expert, but my best guess is that your program tried to create a thread with a stack address that was too high. Remember that with a 2 GB KVA, user processes have only 2 GB to play with instead of 3 GB, so attempting to mmap() a stack above about 2 GB would cause pthread_create() to return EAGAIN. Yes this makes sense, however this call to pthread_create didn't specify any special addresses for the new thread. The pthread_create was called with the NULL attribute which means that the system defaults were being used. Something in the system has gone wrong... I just glanced at the source in -STABLE, and it appears to be a pthreads bug. (Then again, maybe I'm missing something, since nobody seems to have noticed this before.) The default address at which new thread stacks are created is just below the main stack. This address is based on the lexical constant USRSTACK, but it should be initialized in uthread_init() based on the kern.usrstack value returned by sysctl. (The correct value is already used to map the main stack's red zone.) The result is that you need to make world and recompile any apps statically linked against pthreads after building your new kernel in order to get things to work. I don't have time to fiddle with pthreads until after Christmas, but you might see if the following patch (against -STABLE) helps when you reduce the configured KVA size without remaking pthreads. Index: uthread/uthread_init.c === RCS file: /home/ncvs/src/lib/libc_r/uthread/uthread_init.c,v retrieving revision 1.23.2.10 diff -u -r1.23.2.10 uthread_init.c --- uthread/uthread_init.c 2002/10/22 14:44:03 1.23.2.10 +++ uthread/uthread_init.c 2002/12/06 13:41:06 @@ -245,6 +245,8 @@ len = sizeof (int); if (sysctl(mib, 2, _usrstack, len, NULL, 0) == -1) _usrstack = (void *)USRSTACK; + _next_stack = _usrstack - PTHREAD_STACK_INITIAL - + PTHREAD_STACK_DEFAULT - (2 * PTHREAD_STACK_GUARD); /* * Create a red zone below the main stack. All other stacks are * constrained to a maximum size by the paramters passed to To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Fri, 6 Dec 2002, David Schultz wrote: ... Yes this makes sense, however this call to pthread_create didn't specify any special addresses for the new thread. The pthread_create was called with the NULL attribute which means that the system defaults were being used. Something in the system has gone wrong... I just glanced at the source in -STABLE, and it appears to be a pthreads bug. (Then again, maybe I'm missing something, since nobody seems to have noticed this before.) The default address at which new thread stacks are created is just below the main stack. This address is based on the lexical constant USRSTACK, but it should be initialized in uthread_init() based on the kern.usrstack value returned by sysctl. (The correct value is already used to map the main stack's red zone.) The result is that you need to make world and recompile any apps statically linked against pthreads after building your new kernel in order to get things to work. I don't have time to fiddle with pthreads until after Christmas, but you might see if the following patch (against -STABLE) helps when you reduce the configured KVA size without remaking pthreads. ... The patch which you've suggested seems to be logical enough, yet I too cannot understand why nobody got stuck into this thing before. It means that no one in the world ever changed KVA space and used pthreads then, however real life can often be more rich than we think about it. Thank you alot, now I can estimate how this issue can influence the server, there can be nothing worse than some unknown thing which you know is living and messing things up. Now I know which software can be victim to it and what can be considered to be safe. I don't have much opportunity of fiddling with this production server, but if this issue arise again your patch will be handy. BTW will you not be sending it as a bug report with a patch already ready? Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Wed, 4 Dec 2002, Terry Lambert wrote: grep -B 7 KVA_ /sys/i386/conf/LINT -- Terry Thanks a lot Terry, and will you please correct me if I'm wrong, so I don't mess anything up on a production server? The kernel option in question is KVA_PAGES, correct? Because it's not defined in the custom server's kernel then it's value default to 256 (FreeBSD 4.5-STABLE), which makes the KVA space to occupy 1G. Then if I make KVA_PAGES=512 (KVA space 2G), will it solve the problem for this particular server having 4G of phisical memory, or must KVA_PAGES be 768 (KVA space 3G)? Have some other options to be tuned besides KVA_PAGES? Thanks again Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Varshavchick Alexander wrote: On Wed, 4 Dec 2002, Terry Lambert wrote: grep -B 7 KVA_ /sys/i386/conf/LINT Thanks a lot Terry, and will you please correct me if I'm wrong, so I don't mess anything up on a production server? The kernel option in question is KVA_PAGES, correct? Yes. Because it's not defined in the custom server's kernel then it's value default to 256 (FreeBSD 4.5-STABLE), which makes the KVA space to occupy 1G. Then if I make KVA_PAGES=512 (KVA space 2G), will it solve the problem for this particular server having 4G of phisical memory, or must KVA_PAGES be 768 (KVA space 3G)? Have some other options to be tuned besides KVA_PAGES? IMO, KVA need to be more than half of physical memory. But I tend to use a lot of mbufs and mbuf clusters in products I work on lately (mostly networking stuff). If you don't tune kernel memory usage up, then you may be able to get away with 2G. So: 2G might be OK, 3G would be more certain, given you are cranking some things up, in the config you posted, that make me think you will be eating more physical memory. If you follow the 1.5 rule, then you will have 6G of swap when you have 4G of physical RAM, and will definitely need to go for 3G of KVA space. Note, though: all space you add to KVA is subtracted from the process address space, so if you need big processes, sometimes you are better off with less physical RAM, and a larger user virtual address space. For other tuning information, see also man tuning. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Thu, 5 Dec 2002, Terry Lambert wrote: ... Because it's not defined in the custom server's kernel then it's value default to 256 (FreeBSD 4.5-STABLE), which makes the KVA space to occupy 1G. Then if I make KVA_PAGES=512 (KVA space 2G), will it solve the problem for this particular server having 4G of phisical memory, or must KVA_PAGES be 768 (KVA space 3G)? Have some other options to be tuned besides KVA_PAGES? IMO, KVA need to be more than half of physical memory. But I tend to use a lot of mbufs and mbuf clusters in products I work on lately (mostly networking stuff). If you don't tune kernel memory usage up, then you may be able to get away with 2G. So: 2G might be OK, 3G would be more certain, given you are cranking some things up, in the config you posted, that make me think you will be eating more physical memory. Are you talking primarily about SHMMAXPGS=262144 option here? Then may be it'll be oevrall better to reduce it and make KVA space 2G, to leave more room for user address space? You see, there cannot be any much possibility to make experiments with this server, so I very much relay on your advices, thank you again. If you follow the 1.5 rule, then you will have 6G of swap when you have 4G of physical RAM, and will definitely need to go for 3G of KVA space. No, the swap is very slightly used on this server, and the total swap size is 2G. Note, though: all space you add to KVA is subtracted from the process address space, so if you need big processes, sometimes you are better off with less physical RAM, and a larger user virtual address space. For other tuning information, see also man tuning. Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Varshavchick Alexander wrote: So: 2G might be OK, 3G would be more certain, given you are cranking some things up, in the config you posted, that make me think you will be eating more physical memory. Are you talking primarily about SHMMAXPGS=262144 option here? Then may be it'll be oevrall better to reduce it and make KVA space 2G, to leave more room for user address space? That's the one I was referring to, yes, but you didn't post your whole config (please do *NOT* post it; I can't spend the time on going over it line by line). Tuning is a skill; it can be plotted out as a cookbook recipe, but it takes a lot of work to do that, and no one has volunteered. Basically, to write out a cookbook, you have to know where every byte of memory is going in the kernel, and what tunables impact each other, and how they are related. Once you know that, you could easily write a program to kick out a configuration file for various usages, or even modify the code to auto-tune itself (everything by KVA space, which impacts the base address that the kernel gets linked to... unless you compile the entire kernel PIC, which I do not recommend). But knowing the information is hard. I know it for 4.3 and 4.4. You see, there cannot be any much possibility to make experiments with this server, so I very much relay on your advices, thank you again. If you are having system freeses at random, and you want to fix them instead of living with them, some experimentation is going to be inevitable. I don't know enough about your installation to be able to give you a kernel config file to use that will magically fix all your current issues for you, and prevent future issues from coming up. That's going to have to be up to you. If you follow the 1.5 rule, then you will have 6G of swap when you have 4G of physical RAM, and will definitely need to go for 3G of KVA space. No, the swap is very slightly used on this server, and the total swap size is 2G. It doesn't matter. The amount of swap the kernel allocates page tables for is based on the amount of physical RAM in the machine. You pay for the page tables whether you use them or not, for swap, for the kernel, and for any memory which you permit to be allocated at interrupt time, plus any allocations that occur after you are up and running, until you run out of physical RAM. This is one of those things you just have to know about how the kernel uses virtual memory, if you are going to be a skilled kernel tuner. As a rule, swap should be at least physical memory size + 64K on any system that you need to be able to get a system dump from, since it needs to dump physical RAM. If you are not worried about the machine falling over, then you can ignore that. Note that man tuning suggests 2* physical RAM for swap. PS: I am going to be out of touch (able to download, but not send email) for the next couple of days... up to a week. If you have more questions, and they can't wait, you will need to ask someone else. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Thu, 5 Dec 2002, Terry Lambert wrote: IMO, KVA need to be more than half of physical memory. But I tend to use a lot of mbufs and mbuf clusters in products I work on lately (mostly networking stuff). If you don't tune kernel memory usage up, then you may be able to get away with 2G. A question arises. The value 256 (1G KVA space) acts as a default for any system installation, not depending of real phisical memory size. So for any server with RAM less than 2G (which is a majority I presume) the KVA space occupies more than half of physical memory. It can even be more than TOTAL phisical memory for servers with RAM less than 1G. Isn't it bad for a system? It seems that it is not. Then why cannot the KVA space always be made as some big value? If it is important for servers with large RAM, why it is not or a smaller servers? Can anybody besides Terry which seems to be unavailable now help? Regards Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Thus spake Varshavchick Alexander [EMAIL PROTECTED]: A question arises. The value 256 (1G KVA space) acts as a default for any system installation, not depending of real phisical memory size. So for any server with RAM less than 2G (which is a majority I presume) the KVA space occupies more than half of physical memory. It can even be more than TOTAL phisical memory for servers with RAM less than 1G. Isn't it bad for a system? It seems that it is not. Then why cannot the KVA space always be made as some big value? If it is important for servers with large RAM, why it is not or a smaller servers? In FreeBSD, each process has a unique 4G virtual address space associated with it. Not every virtual page in every address space has to be associated with real memory. Most pages can be pushed out to disk when there isn't enough free RAM, and unallocated parts of the virtual address space aren't backed by anything. (Referencing an unmapped page that the system doesn't know about generally causes the program or OS to crash. You've probably seen these as ``segmentation faults'' and ``page fault in kernel mode'' panics.) To simplify things, the kernel is mapped into a fixed location in every address space. The KVA parameter controls how big a chunk the kernel gets; the remainder goes to user processes. However, only the part of the KVA reservation that the kernel actually uses is wired to physical memory. For example, if you have a 1 GB KVA reservation and the kernel allocates only 20 MB of RAM, then only 20 MB of RAM is needed (plus some epsilon if you want to be picky), but in theory, the kernel could allocate and manage up to 1 GB of data. You don't lose extra physical memory for increasing KVA, but a large KVA size does constrain the virtual address space available to user processes. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Thu, 5 Dec 2002, Varshavchick Alexander wrote: On Thu, 5 Dec 2002, Terry Lambert wrote: IMO, KVA need to be more than half of physical memory. But I tend to use a lot of mbufs and mbuf clusters in products I work on lately (mostly networking stuff). If you don't tune kernel memory usage up, then you may be able to get away with 2G. A question arises. The value 256 (1G KVA space) acts as a default for any system installation, not depending of real phisical memory size. So for any server with RAM less than 2G (which is a majority I presume) the KVA space occupies more than half of physical memory. It can even be more than TOTAL phisical memory for servers with RAM less than 1G. Isn't it bad for a system? It seems that it is not. Then why cannot the KVA space always be made as some big value? If it is important for servers with large RAM, why it is not or a smaller servers? Can anybody besides Terry which seems to be unavailable now help? It controls the split between virtual address space, not allocation of physical memory. If KVA is turned up to 3GB (say) then userland virtual address space for all processes is limited to 1GB (each). For Terry's stuff (networking, mostly, and probably mostly in the kernel anyway) this is beneficial. For (to pick an example at random) anyone running java* or other large userland processes, having only 1GB of elbow-space (physical or virtual) is often not sufficient. jan * You've plenty of resources and an infinite number of threads are bound to make fair progress seem to be a summation of the java way :-) -- jan grant, ILRT, University of Bristol. http://www.ilrt.bris.ac.uk/ Tel +44(0)117 9287088 Fax +44 (0)117 9287112 http://ioctl.org/jan/ Work #90: As many pseudo-intellectual sycophants as necessary to make one inarticulate scotsman think he's a genius in command of The Profound. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Thu, 5 Dec 2002, David Schultz wrote: In FreeBSD, each process has a unique 4G virtual address space associated with it. Not every virtual page in every address space has to be associated with real memory. Most pages can be pushed out to disk when there isn't enough free RAM, and unallocated parts of the virtual address space aren't backed by anything. (Referencing an unmapped page that the system doesn't know about generally causes the program or OS to crash. You've probably seen these as ``segmentation faults'' and ``page fault in kernel mode'' panics.) To simplify things, the kernel is mapped into a fixed location in every address space. The KVA parameter controls how big a chunk the kernel gets; the remainder goes to user processes. However, only the part of the KVA reservation that the kernel actually uses is wired to physical memory. For example, if you have a 1 GB KVA reservation and the kernel allocates only 20 MB of RAM, then only 20 MB of RAM is needed (plus some epsilon if you want to be picky), but in theory, the kernel could allocate and manage up to 1 GB of data. You don't lose extra physical memory for increasing KVA, but a large KVA size does constrain the virtual address space available to user processes. Thank you David for such an excellent explanation. So if sysctl reports vm.zone_kmem_pages: 5413 vm.zone_kmem_kvaspace: 218808320 vm.kvm_size: 1065353216 vm.kvm_free: 58720256 does it mean that total KVA reservation is 1065353216 bytes (1G) and almost all of it is really mapped to physical memory because only 58720256 (56M) is free, and the server is balancing on the edge of crashing with KVA going out? Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Wed, 4 Dec 2002, Terry Lambert wrote: Marc Recht wrote: Every now and this I hear people saying (mostly you :)) that some problems are KVA related or that the KVA must be increased. This makes me a bit curious, since I've never seen problems like that on Linux. It sounds for me, the not kernel hacker, a bit like something which should be set at boot time (or via sysctl). Have you got some pointers which explain FreeBSD's KVA ? I have written documentation for FreeBSD 4.3/4.4. Unfortunately, everyone keeps substituting activity for action, and hacking away at the code, so it doesn't sit still long enough to match any useful documentation; otherwise, I would have published what I wrote in Pentad Embedded Systems Journal already (example: the ^^^ I appreciate some of the info you give. But every time you reference a proper noun (person, journal, etc.), Google only gives results of you talking about it in FreeBSD list archives! See also freebsd mitre netbeui What kind of conclusion is one to draw from that? -Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Varshavchick Alexander wrote: Can it be so that kernel maxusers=768 value being more than 512 leads to spontaneous system freezes which can take up to several hours when the system is just sleeping (only replying to ping) and doing nothing else, not allowing to telnet or anything. The system is 4.5-STABLE with much RAM (4 Gb) and the box has a heavy enough traffic so a bunch of other kernel options have been increased: [ ... settings ... ] With these settings, and that much physical RAM, you should set your KVA space to 3G (the default is 2G); have you? Most likely, you are running out of KVA space for mappings. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
On Wed, 4 Dec 2002, Terry Lambert wrote: Varshavchick Alexander wrote: Can it be so that kernel maxusers=768 value being more than 512 leads to spontaneous system freezes which can take up to several hours when the system is just sleeping (only replying to ping) and doing nothing else, not allowing to telnet or anything. The system is 4.5-STABLE with much RAM (4 Gb) and the box has a heavy enough traffic so a bunch of other kernel options have been increased: [ ... settings ... ] With these settings, and that much physical RAM, you should set your KVA space to 3G (the default is 2G); have you? Most likely, you are running out of KVA space for mappings. No, I didn't do it, and I'm not sure how to perform it, can you please advise? Thanks a lot! -- Terry Alexander Varshavchick, Metrocom Joint Stock Company Phone: (812)118-3322, 118-3115(fax) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Varshavchick Alexander wrote: With these settings, and that much physical RAM, you should set your KVA space to 3G (the default is 2G); have you? Most likely, you are running out of KVA space for mappings. No, I didn't do it, and I'm not sure how to perform it, can you please advise? Thanks a lot! grep -B 7 KVA_ /sys/i386/conf/LINT -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
With these settings, and that much physical RAM, you should set your KVA space to 3G (the default is 2G); have you? Most likely, you are running out of KVA space for mappings. Every now and this I hear people saying (mostly you :)) that some problems are KVA related or that the KVA must be increased. This makes me a bit curious, since I've never seen problems like that on Linux. It sounds for me, the not kernel hacker, a bit like something which should be set at boot time (or via sysctl). Have you got some pointers which explain FreeBSD's KVA ? Regards, Marc Premature optimization is the root of all evil. -- Donald E. Knuth To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: maxusers and random system freezes
Marc Recht wrote: Every now and this I hear people saying (mostly you :)) that some problems are KVA related or that the KVA must be increased. This makes me a bit curious, since I've never seen problems like that on Linux. It sounds for me, the not kernel hacker, a bit like something which should be set at boot time (or via sysctl). Have you got some pointers which explain FreeBSD's KVA ? I have written documentation for FreeBSD 4.3/4.4. Unfortunately, everyone keeps substituting activity for action, and hacking away at the code, so it doesn't sit still long enough to match any useful documentation; otherwise, I would have published what I wrote in Pentad Embedded Systems Journal already (example: the KVA_PAGES stuff came in after FreeBSD 4.3/4.4, and blew out two paragraphs on what to modify where, and how to calculate the values to use). The best documentation is probably Matt Dillon's article in Daemon News, the FreeBSD Developer's handbook, or the German guy's article in English (sorry for not remembering your name), depending on what part of things you are interested in. If you could get people to leave the damn code alone for a while, I'd be willing to update my article to FreeBSD RELENG_4 (-STABLE), and publish it. One of the major problems with undocumented code is that weenies are unwilling to sit down and understand it, so they rewrite it to understand it, instead, and then you are still without documentation. Documentation that's almost right is unbelievably worse than no documentation at all. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
