Re: Is better one or more Tomcat instances per machine
Unless you have real memory requirements, one tomcat instance is better, at least in terms of maintenance. There is no real advantage in multi-instancing. But the limit for max memory you can effectively use in your java program on a 32bit linux, i assume you use (same for windows), lies far below 2Gb. If you need more ram you should consider upgrading to a 64 bit OS / VM. Leon On 3/14/07, Wenca [EMAIL PROTECTED] wrote: Hi all, I am now preparing the deployment scenario for our new webapp and I am not sure if it is better to have one Tomcat instance per server machine or to have more instances. I have 3 servers - dual Intel Xeon 3GHz, 4GB RAM each (about 3GB is available for Tomcat etc.). There will be running only about 2-3 webapps. No need to separate webaps to different JVMs. Servers will be in a cluster behind a load balancer. In fact it will be clustered probably with Open Terracotta sessions. So, is it better (due to performance and number of concurrently served users) to have one Tomcat instance per machine, running in JVM with 2GB heap size (I know this is limit on 32bit system) or to have 2 TC instances (say 1.3GB each) or 3 TC inst. (0.9GB each), 4 inst., . Thank you very much Wenca - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Leon, Leon Rosenberg wrote: But the limit for max memory you can effectively use in your java program on a 32bit linux, i assume you use (same for windows), lies far below 2Gb. I've been trying to find the real nature of this memory limit. I have never experienced this limit, but the only time I've required a heap larger than a gig or so was at an old job, and I don't recall exactly what the deal was. The reading I've done so far on this subject leads me to believe that most people don't know what they heck they're talking about. Some claim that 32-bit OSs can't use more than 4GB RAM (they can) or that they have 2GB/2GB kernel and process memory boundaries (they don't, except that I think MS Windows might have this), or that the problem is that 32-bit OSs cannot allocate more than 2GB of contiguous memory (which may be true, but seems irrelevant, since the Java heap need not be contiguous). So, can you point me to a resource that actually explains what the problem is and why it exists? Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF9/kh9CaO5/Lv0PARAlAGAKCaFqOemTtz+odAKrgWDnujkcinlgCeOn10 8yeqOiR/saXW01YjxoaXnoM= =5h/W -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
En l'instant précis du 14/03/07 14:31, Christopher Schultz s'exprimait en ces termes: Leon, Leon Rosenberg wrote: But the limit for max memory you can effectively use in your java program on a 32bit linux, i assume you use (same for windows), lies far below 2Gb. I've been trying to find the real nature of this memory limit. I have never experienced this limit, but the only time I've required a heap larger than a gig or so was at an old job, and I don't recall exactly what the deal was. The reading I've done so far on this subject leads me to believe that most people don't know what they heck they're talking about. Some claim that 32-bit OSs can't use more than 4GB RAM (they can) or that they have 2GB/2GB kernel and process memory boundaries (they don't, except that I think MS Windows might have this), or that the problem is that 32-bit OSs cannot allocate more than 2GB of contiguous memory (which may be true, but seems irrelevant, since the Java heap need not be contiguous). So, can you point me to a resource that actually explains what the problem is and why it exists? Thanks, -chris 32 bits architecture, a memory pointer is 32 bits and thus can only address memory ranges between 0 to 2^32, that makes 4G back in kernel 2.4 time, you add same pointer to address kernel space from within application and application allocated space. However, there needed to be a way to distinguish easily between kernel space and application space and by this mean prevent user space to play in kernel space. This was decided the separation would be at 2G. In those time this was not a real problem. My home computer had barely 32M memory, so why would i need to address more than 2G? But progressively this limit became a problem to user applications. Patch were issued to do a 3G/1G split but it was not enough. This has changed. An new architecture was brought in CPU (at pentium II time?) that allowed OS to do a 4G/4G mapping in 32 bits mode. Since you don't access kernel space from user mode directly, you can simply use different pointer for kernel space and user space. Hardware wiring allows to map to different addresses. At application level, it does not change anything, you still do a malloc and use resulting pointer. At kernel level, changes have been made when datas are transfered from kernel space to user space. However, for this 4G/4G split to work, you will need a 2.6 kernel i think. So, 4G is the maximum memory any application can allocate in 32 bits modes, be it at one time or in several operations. At least that's how i understood story. I only have 1G ram +1G swap, i can't test, but it bet using a kernel 2.6 you can use 4G for your jvm... Also, you can go 64bits if it's not enough :) - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: [OT]Re: Is better one or more Tomcat instances per machine The below applies only to 32-bit systems, of course. Some claim that 32-bit OSs can't use more than 4GB RAM Lots of people seem to confuse virtual space with real memory. There is a 4GB limitation on virtual space per process (IA32 architecture), but the real memory (RAM) limit depends on the OS and the number of address lines implemented between the CPU and memory, up to the max 64 GB supported by IA32. that they have 2GB/2GB kernel and process memory boundaries Windows certainly does have such a boundary (although in certain server versions, this can be changed to 3GB/1GB user/kernel). Most other OSs don't have a fixed boundary, but they do have a maximum virtual memory size (set by ulimit in Linux), and each installation can specify a default. since the Java heap need not be contiguous Wouldn't it be nice if that were true. I haven't looked at the Java SE 6 HotSpot code yet, but in all prior versions of Sun's JVM, the heap was allocated as one contiguous chunk, including the PermGen. HotSpot has the ability to shift the boundaries between generations, and thus requires the heap to be contiguous. Windows makes the situation worse by scattering various DLLs and other allocations throughout the 2GB user process space, thereby significantly reducing the contiguous space available for the JVM heap. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14/03/2007, at 2:31 PM, Christopher Schultz wrote: The reading I've done so far on this subject leads me to believe that most people don't know what they heck they're talking about. Some claim that 32-bit OSs can't use more than 4GB RAM (they can) or that they have 32bit OSes can not use more than 4GB RAM. What you are probably referring to is PAE, and there the kernel splits the 'extra' memory into chunks, and can give each process part of this chunk - a single process however, under linux can not use more than 2GB (or 3GB) of RAM (depending on how the kernel was compiled) 2GB/2GB kernel and process memory boundaries (they don't, except that I think MS Windows might have this), or that the problem is that 32-bit Well think again http://support.microsoft.com/kb/283037 It will tell you all about the 2G/ 3G issues as well. So, can you point me to a resource that actually explains what the problem is and why it exists? The above link should hopefully be enough. Cheers Andrew -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFF+APdW126qUNSzvURAqUtAJ0eNCOTgjU+/s0eZ+lGpcq2nNyu1wCfZj4u 7UAUSZz4U9SjvFHYAMMBp0w= =KRln -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Is better one or more Tomcat instances per machine
From: Leon Rosenberg [mailto:[EMAIL PROTECTED] There is no real advantage in multi-instancing. A minor advantage is that if you allocate one webapp per container, if one webapp fails it only takes down its own container. Well-coded webapps should never cause this - and, of course, we all manage completely bug-free webapps all the time, don't we? :-) - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14/03/2007, at 3:11 PM, David Delbecq wrote: This has changed. An new architecture was brought in CPU (at pentium II time?) that allowed OS to do a 4G/4G mapping in 32 bits mode. Since you don't access kernel space from user mode directly, you can simply use different pointer for kernel space and user space. Hardware wiring allows to map to different addresses. At application level, it does not change anything, you still do a malloc and use resulting pointer. At kernel level, changes have been made when datas are transfered from kernel space to user space. However, for this 4G/4G split to work, you will need a 2.6 kernel i think. So, 4G is the maximum memory any application can allocate in 32 bits modes, be it at one time or in several operations. Ummm - 4G/4G can't work - I think what you are referring to is PAE, and this does what effectively was earlier called bank switching - oh those memorys of TRS80s and Commodore 64s! (Oh and this works with 2.4) Cheers Andrew -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFF+AS1W126qUNSzvURAsx/AJ0bOtkSMX32G4Iu6Mfa/2i6F2UxGgCdEvPi 9sN8iQ0+WYNuYWAhv+ue4es= =nKSG -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Andrew Miehs [mailto:[EMAIL PROTECTED] 32bit OSes can not use more than 4GB RAM. What you are probably referring to is PAE, and there the kernel splits the 'extra' memory into chunks, and can give each process part of this chunk - a single process however, under linux can not use more than 2GB (or 3GB) of RAM (depending on how the kernel was compiled) Let's be clear about the distinction between OS and process managed by OS: - The OS as a whole can manage 4 Gbytes of physical memory using PAE; - On some OSs (Linux, perhaps?), a user process cannot be allocated 4 Gbytes of RAM; - On other OSs (Windows), a user process *can* be allocated 4 Gbytes of RAM. Microsoft SQL Server (2000 Enterprise and up) use the facilities built into Windows 2000 and up to allocate PAE memory to the sinle SQL Server process. - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Is better one or more Tomcat instances per machine
On 14/03/2007, at 3:17 PM, Peter Crowther wrote: From: Leon Rosenberg [mailto:[EMAIL PROTECTED] There is no real advantage in multi-instancing. A minor advantage is that if you allocate one webapp per container, if one webapp fails it only takes down its own container. Well-coded webapps should never cause this - and, of course, we all manage completely bug-free webapps all the time, don't we? :-) Actually, according to the JSP spec, one Webapp is not allowed to effect the other, so if you use Tomcat, one Webapp can not take out the other... :-) Duck and run for cover Cheers Andrew - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
On 14/03/2007, at 3:21 PM, Peter Crowther wrote: Let's be clear about the distinction between OS and process managed by OS: - The OS as a whole can manage 4 Gbytes of physical memory using PAE; - On some OSs (Linux, perhaps?), a user process cannot be allocated 4 Gbytes of RAM; Sorry, was being lazy, yes a 32bit OS can via PAE address more than 4GB of memory SPACE. On linux, and other *nixes running on IA32 hardware, you will find that you have User and Kernel Space. The user space is the amount of RAM you as a process can allocate for this single process. It is usually a 2G/2G split, although you will sometimes find 3/1 or 1/3. This is because there needs to be some way for your process to interact with the hardware and kernel. - On other OSs (Windows), a user process *can* be allocated 4 Gbytes of RAM. Microsoft SQL Server (2000 Enterprise and up) use the facilities built into Windows 2000 and up to allocate PAE memory to the sinle SQL Server process. Hey cool, looks as if Microsoft has implemented bank switching for user space, although still rather primitive. (AWE) Here a few links http://support.microsoft.com/kb/274750 http://msdn2.microsoft.com/en-us/library/ms190673.aspx You learn something new every day! Cheers Andrew - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David, David Delbecq wrote: 32 bits architecture, a memory pointer is 32 bits and thus can only address memory ranges between 0 to 2^32, that makes 4G back in kernel 2.4 time Pointers didn't get bigger in 2.6, so the 4GB process limit is still there in 32-bit 2.6 linux kernels. This has changed. An new architecture was brought in CPU (at pentium II time?) that allowed OS to do a 4G/4G mapping in 32 bits mode. I don't think this has anything to do with hardware. Apparently, certain 32-bit versions of Solaris do not have this problem, and Java can use as much memory as any process. So, 4G is the maximum memory any application can allocate in 32 bits modes, be it at one time or in several operations. Right, but we're talking about a 2GB limit. At least that's how i understood story. I only have 1G ram +1G swap, i can't test, but it bet using a kernel 2.6 you can use 4G for your jvm... Nope: [EMAIL PROTECTED] ~/projects/diagnosis $ uname -a Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005 i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux [EMAIL PROTECTED] ~/projects/diagnosis $ java -Xms1732M -Xmx1732M JDBCTest Error occurred during initialization of VM Could not reserve enough space for object heap If I decrease the heap size by 1M, it works. shrug - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+Av89CaO5/Lv0PARApLQAKCsfJClSdo9EgqR3ME7sw3NNxibsQCdEPFS HWANdJSSiKM4awlb1TcoqGo= =DERq -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Andrew Miehs [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine The user space is the amount of RAM you as a process can allocate for this single process. No - RAM has nothing to do with the split. Process memory is the amount of virtual space allocated to the user; the OS may choose to use much less RAM behind that virtual space. Page thrashing may occur, but the process still gets to play with what it thinks is 2GB of memory. This is because there needs to be some way for your process to interact with the hardware and kernel. The global and local descriptor tables and the page directories/tables provide the translation between virtual space and RAM. There need not be any split between user space and kernel space, but it's useually more efficient to create such a boundary. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, I knew you'd come through. It's always nice to have a VM hacker around for questions like this. Caldarale, Charles R wrote: that they have 2GB/2GB kernel and process memory boundaries Windows certainly does have such a boundary (although in certain server versions, this can be changed to 3GB/1GB user/kernel). Most other OSs don't have a fixed boundary, but they do have a maximum virtual memory size (set by ulimit in Linux), and each installation can specify a default. Hmm perhaps I have a virtual memory limit. I have 1GB of physical RAM. While allocating a 3GB heap is pretty stupid for me, I still ought to be able to do it. My ulimit man page sucks. Anyone know how to change the virtual memory limits on Linux 2.6? since the Java heap need not be contiguous Wouldn't it be nice if that were true. I haven't looked at the Java SE 6 HotSpot code yet, but in all prior versions of Sun's JVM, the heap was allocated as one contiguous chunk, including the PermGen. HotSpot has the ability to shift the boundaries between generations, and thus requires the heap to be contiguous. Really? That sounds hard to believe. If the heap can grow and shrink, then how can the VM hope to get one bigass memory chunk for the heap? If the heap goes from 512MB to 768MB and the existing heap can't grow that far (in a single heap-top-move operation), does the VM freeze everything and relocate the heap to another section of memory? That doesn't seem right. The fact that the boundaries can be moved really complicates things. I had always assumed that the heap map that most people see in pictures was only to make it simpler to understand. I thought that the heaps were located wherever they needed to be. Oh, well. Still, the only remaining reason that 2GB+ wouldn't work is because of a (potential) 2GB contiguous memory allocation restriction. I see no such restriction in the malloc or brk man pages on my system. What gives? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+A4A9CaO5/Lv0PARAtIyAJ97F8a5K+DmfNrstvtEruFdRyXdrwCdEiFa j3AguINPMHBGDwuWSojuTXE= =WK+n -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine I don't think this has anything to do with hardware. It does. To quote from the IA32 architecture spec: Starting with the Pentium Pro processor, the IA-32 architecture also supports an extension of the physical address space to 236 bytes (64 GBytes); with a maximum physical address of FH. This extension is invoked in either of two ways: * Using the physical address extension (PAE) flag, located in bit 5 of control register CR4. * Using the 36-bit page size extension (PSE-36) feature (introduced in the Pentium III processors). Apparently, certain 32-bit versions of Solaris do not have this problem, and Java can use as much memory as any process. Not sure what the last clause above means, but there are differences in Solaris on Sparc and Solaris on x86. [EMAIL PROTECTED] ~/projects/diagnosis $ uname -a Linux chadis 2.6.14-gentoo-r5 #2 PREEMPT Sat Dec 17 16:30:55 EST 2005 i686 AMD Athlon(tm) XP 1700+ AuthenticAMD GNU/Linux [EMAIL PROTECTED] ~/projects/diagnosis $ java -Xms1732M -Xmx1732M JDBCTest Error occurred during initialization of VM Could not reserve enough space for object heap If I decrease the heap size by 1M, it works. shrug What are your ulimit settings? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andrew, Andrew Miehs wrote: 32bit OSes can not use more than 4GB RAM. ??! A process on a 32-bit OS can't use more than 4GB of RAM, but the OS certainly can. 2GB/2GB kernel and process memory boundaries (they don't, except that I think MS Windows might have this), or that the problem is that 32-bit Well think again http://support.microsoft.com/kb/283037 It will tell you all about the 2G/ 3G issues as well. Note that I said that MS Windows has it's own issues. This is not endemic to 32-bit OSs in general, though. The OP was using Linux, so I'm basically ignoring MS Windows, anyway. So, can you point me to a resource that actually explains what the problem is and why it exists? The above link should hopefully be enough. Not really. There's not a lot of Linux information on Microsoft's site :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+A659CaO5/Lv0PARAhqEAKCFT3nEmnvFkFzO7BW8nr+KOTG16QCfQaAs eBTTJkT06Hc3NO+fnNoQ78w= =wmBe -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
1) We are using tomcat in production with -Xmx1524M up to 1700, but I have never used it with -Xmx more than 2G. I can also verify that top shows java process using more than 1GB. Java process shows 200MB more allocated memory than -Xmx in top. 2) I have CentOS 4.4 machine at home that has 4GB of ram, I will test it as soon as I get some time. Andrew Pliszka Caldarale, Charles R wrote: From: Andrew Miehs [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine The user space is the amount of RAM you as a process can allocate for this single process. No - RAM has nothing to do with the split. Process memory is the amount of virtual space allocated to the user; the OS may choose to use much less RAM behind that virtual space. Page thrashing may occur, but the process still gets to play with what it thinks is 2GB of memory. This is because there needs to be some way for your process to interact with the hardware and kernel. The global and local descriptor tables and the page directories/tables provide the translation between virtual space and RAM. There need not be any split between user space and kernel space, but it's useually more efficient to create such a boundary. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14/03/2007, at 3:52 PM, Caldarale, Charles R wrote: The user space is the amount of RAM you as a process can allocate for this single process. No - RAM has nothing to do with the split. Process memory is the amount of virtual space allocated to the user; the OS may choose to use much less RAM behind that virtual space. Page thrashing may occur, but the process still gets to play with what it thinks is 2GB of memory. Arrgh! Ok - RAM was the wrong word. This is because there needs to be some way for your process to interact with the hardware and kernel. The global and local descriptor tables and the page directories/tables provide the translation between virtual space and RAM. There need not be any split between user space and kernel space, but it's useually more efficient to create such a boundary. Your kernel, and the things which are doing your process switching need somewhere to run - if you switch them out of your 4GB of virtual address space, how are they ever meant to 'come back to life' on the next context switch. As for it 'NEEDING' to be 3/1 or 2/2 - agreed - but some amount does need to be there. (This is all of course assuming you are running some sort of preemptive operating system) Didn't the reason for choosing this size have something to do with the memory required by the PCI slots, etc? Andrew -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFF+BA+W126qUNSzvURAhAlAJ4qrZ1PEiQ3wnNwRBsY8lOfbaBbFACgieQD uLK9GNTvsS7zbo1weFpM9jk= =4UIv -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine Hmm perhaps I have a virtual memory limit. I have 1GB of physical RAM. While allocating a 3GB heap is pretty stupid for me, I still ought to be able to do it. If your OS will allow, yes you should. You may need to increase your swap file space, and you may burn out your hard drive thrashing pages in and out. My ulimit man page sucks. Anyone know how to change the virtual memory limits on Linux 2.6? Here's a link for SUSE: http://en.opensuse.org/SDB:Ulimit_for_Protection_Against_%22Out_of_memor y%22 If the heap can grow and shrink, then how can the VM hope to get one bigass memory chunk for the heap? The JVM reserves - but does not commit - the -Xmx size during initialization. If that's not available, the JVM won't start. Once allocated, commits and decommits occur to expand and shrink the in-use size. does the VM freeze everything and relocate the heap to another section of memory? No, since the maximum size is allocated from the beginning. The fact that the boundaries can be moved really complicates things. Agreed. I think HotSpot has gone a bit overboard in this regard. Still, the only remaining reason that 2GB+ wouldn't work is because of a (potential) 2GB contiguous memory allocation restriction. I see no such restriction in the malloc or brk man pages on my system. What gives? Those are typically generic descriptions aimed at application programmers. What we're discussing here is OS internals, and you'll rarely see those documented anywhere. If you're lucky, the OS vendor will have such limitiations squirreled away in some appendix to its installation guide. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Andrew Miehs [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine Your kernel, and the things which are doing your process switching need somewhere to run - if you switch them out of your 4GB of virtual address space, how are they ever meant to 'come back to life' on the next context switch. In IA32, it is theoretically possible to run every user process with a 4 GB local descriptor table, and when a kernel service is required, the hardware switches to the global descriptor table due to the interrupt; the global one maps the kernel space. Once in kernel mode, the OS code can juggle the segment and page tables any way it sees fit. I don't know if anyone actually does run with a 4 GB user process space; most use a split in the virtual space to avoid the above juggling. Didn't the reason for choosing this size have something to do with the memory required by the PCI slots, etc? Not sure what you mean by this size, since we're discussing lots of them here. There is considerable address space history to look back on, starting with the original IBM PC running in what's now called x86 real mode. Many of the IA32 architecture aspects have evolved with compatibility and migration in mind, so it's not what you'd design from scratch. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Chuck,
Caldarale, Charles R wrote:
From: Christopher Schultz [mailto:[EMAIL PROTECTED]
Hmm perhaps I have a virtual memory limit. I have 1GB
of physical RAM. While allocating a 3GB heap is pretty
stupid for me, I still ought to be able to do it.
If your OS will allow, yes you should. You may need to increase your
swap file space, and you may burn out your hard drive thrashing pages in
and out.
I think I just did using this little beauty:
#include stdio.h
#include stdlib.h
/* 50 MB */
#define CHUNK_SIZE (1024 * 1024 * 50)
int main(int argc, char *arvg[])
{
void *ptr;
int i;
unsigned long bytes = 0l;
for( ; ;) {
ptr = calloc((size_t)CHUNK_SIZE, (size_t)1);
bytes += CHUNK_SIZE;
/* write to our new memory */
for(i=0; iCHUNK_SIZE; ++i)
*((char*)ptr + i) = (char)i;
printf(Heap has now allocated %lu bytes\n, bytes);
if(bytes (unsigned long)(1024 * 1024 * 1024 * 3.9))
fgetc(stdin);
}
}
Oddly enough, it stopped printing stuff out after 1677721600 bytes.
Before I added the write to our memory loop, it ran very quickly and
completed (well, the fgetc was to pause so I could observe memory
usage). I guess that Linux not only does optimistic malloc, but also
optimistic calloc as well. I had hoped that zeroing-out the memory would
count as a write, but apparently it does not.
Anyhow, I believe I'm running out of swap in this situation. I think I
only have a gig of swap on this development machine, which means that I
have a hard limit of 2GB anyway.
Anyone care to try this program on their many-gigabyted development box?
My ulimit man page sucks. Anyone know how to change the virtual memory
limits on Linux 2.6?
Here's a link for SUSE:
http://en.opensuse.org/SDB:Ulimit_for_Protection_Against_%22Out_of_memor
y%22
Thanks for the link. Probably won't help me with my 1GB of virtual
memory. Heh.
If the heap can grow and shrink, then how can the VM hope to
get one bigass memory chunk for the heap?
The JVM reserves - but does not commit - the -Xmx size during
initialization. If that's not available, the JVM won't start. Once
allocated, commits and decommits occur to expand and shrink the in-use
size.
Aah, I see. So the heap is actually fixed (to the maximum value) but the
internal management of the heap will fluctuate.
Still, the only remaining reason that 2GB+ wouldn't work is
because of a (potential) 2GB contiguous memory allocation
restriction. I see no such restriction in the malloc or brk
man pages on my system. What gives?
Those are typically generic descriptions aimed at application
programmers. What we're discussing here is OS internals, and you'll
rarely see those documented anywhere. If you're lucky, the OS vendor
will have such limitiations squirreled away in some appendix to its
installation guide.
Well, since this is Linux, there really isn't an operating guide. I
suppose the man pages couldn't include that information, anyway, since
it's a kernel setting chosen at kernel build-time. But, the GNU malloc
man page does describe the OOM killer behavior exhibited by Linux
kernels (under certain configurations). I had expected that this other
issue might have been covered as well.
I still can't find any documentation about a 2GB contiguous allocation
limit for Linux anywhere. Is this just common knowledge that nobody
has bothered to verify?
I mean, this is entirely academic at this point, but it would be nice to
actually know the truth instead of everyone just posting their 5th-hand
repeated guesses.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFF+BYu9CaO5/Lv0PARAhZGAKC1u/BtCWSdI6eIGeBuH5qXtD1JZQCfTTPw
DaJRF41SVbVFJMBgfozB3lA=
=qd9B
-END PGP SIGNATURE-
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine I don't think this has anything to do with hardware. It does. To quote from the IA32 architecture spec: Sorry... I wasn't being clear. I meant that any splits and/or memory mappings were not enabled by hardware changes. A 32-bit OS could always use linearly accessed memory up to 4GB without any artificial split. Hardware changes might have made it possible to go beyond 4GB (for the OS) and to offer more exotic artificial memory splits (4GB/4GB, for instance). Apparently, certain 32-bit versions of Solaris do not have this problem, and Java can use as much memory as any process. Not sure what the last clause above means, but there are differences in Solaris on Sparc and Solaris on x86. Certainly. This evidence was anecdotal, and I'm not sure if that particular poster (another list) mentioned their config. Something like I use 32-bit Solaris and have no such problem. The fact remains that you can't allocate a VM heap bigger than around 1750MB on my 32-bit, 2.6 Linux kernel. Why not? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+Bas9CaO5/Lv0PARAtXxAKCT6raQLkP8qQcOQzeNM+7r8BTHIwCfQfi3 Gu5ODBHntjJD6Nmm4AujklA= =bFH+ -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Is better one or more Tomcat instances per machine
Heh - ask Murphy about that :) just spawn a thread set priority high and loop forever. At 10:23 3/14/2007, you wrote: On 14/03/2007, at 3:17 PM, Peter Crowther wrote: From: Leon Rosenberg [mailto:[EMAIL PROTECTED] There is no real advantage in multi-instancing. A minor advantage is that if you allocate one webapp per container, if one webapp fails it only takes down its own container. Well-coded webapps should never cause this - and, of course, we all manage completely bug-free webapps all the time, don't we? :-) Actually, according to the JSP spec, one Webapp is not allowed to effect the other, so if you use Tomcat, one Webapp can not take out the other... :-) Duck and run for cover Cheers Andrew - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine I guess that Linux not only does optimistic malloc, but also optimistic calloc as well. I had hoped that zeroing-out the memory would count as a write, but apparently it does not. I'm somewhat suprised, since calloc() normally doesn't know if it's getting fresh memory (guaranteed to be zero) or reusing available memory. Perhaps calloc() on your platform knows when it's expanding the C-heap and therefore avoids the redundant clear. I think I only have a gig of swap on this development machine You should be able to change that with yast or its equivalent. I still can't find any documentation about a 2GB contiguous allocation limit for Linux anywhere. I don't know that there is one. Most of the contiguity problems are with Windows, due to its scattering of DLLs in the user process space. There should be tools available for Linux (there are for Windows) to show what's allocated where in your process space. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, Caldarale, Charles R wrote: From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine I guess that Linux not only does optimistic malloc, but also optimistic calloc as well. I had hoped that zeroing-out the memory would count as a write, but apparently it does not. I'm somewhat suprised, since calloc() normally doesn't know if it's getting fresh memory (guaranteed to be zero) or reusing available memory. Perhaps calloc() on your platform knows when it's expanding the C-heap and therefore avoids the redundant clear. I think that glibc and/or the memory manager is just magic. ;) It already knows whether non-glibc code modified the memory, so perhaps it also knows that when you actually try to use a page of memory created with calloc, that it should initialize it to zero when you use it, not when you allocate it. I figured the opposite, but my tests appear to support a zero-on-use implementation. I think I only have a gig of swap on this development machine You should be able to change that with yast or its equivalent. mkswap works, too. I just don't have any unallocated disk space. Ooh, maybe I can put my swap on a ramdisk... I still can't find any documentation about a 2GB contiguous allocation limit for Linux anywhere. I don't know that there is one. Most of the contiguity problems are with Windows, due to its scattering of DLLs in the user process space. There should be tools available for Linux (there are for Windows) to show what's allocated where in your process space. There are, I just don't know how to interpret the information they display ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+CBG9CaO5/Lv0PARAn7XAJ9mIpGwEwlWsQQYAsBXw8ynl3cd4QCgmHwa nNo8YespEQJXcd7z335UVgk= =0ppt -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine The fact remains that you can't allocate a VM heap bigger than around 1750MB on my 32-bit, 2.6 Linux kernel. Why not? You have to wait for a full moon... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
| From: Christopher Schultz [mailto:[EMAIL PROTECTED] | Sent: Wednesday, 14 March, 2007 10:37 | | The fact remains that you can't allocate a VM heap bigger than around | 1750MB on my 32-bit, 2.6 Linux kernel. Why not? If, as you stated earlier, you only have 1G of physical and 1G of virtual memory, then that's probably the limit you're hitting. I don't know about Linux, but under Solaris you can create a swap file anywhere, even on a regular filesystem (under NetBSD you had to put them on their own disk partitions). Maybe for the sake of experimenting you could use a USB flash drive for a swap device. Even if it's just 256MB, you should be able to see if it's a swap space limitation or something more sinister... - The information contained in this message is confidential proprietary property of Nelnet, Inc. and its affiliated companies (Nelnet) and is intended for the recipient only. Any reproduction, forwarding, or copying without the express permission of Nelnet is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this e-mail. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tracy, Nelson, Tracy M. wrote: | From: Christopher Schultz [mailto:[EMAIL PROTECTED] | Sent: Wednesday, 14 March, 2007 10:37 | | The fact remains that you can't allocate a VM heap bigger than around | 1750MB on my 32-bit, 2.6 Linux kernel. Why not? If, as you stated earlier, you only have 1G of physical and 1G of virtual memory, then that's probably the limit you're hitting. Perhaps, but the JVM actually refuses to start right away. In my eat all my memory tests, I was able to eat around 1.6GB before I brought my machine to a crawl. It took more than a minute for my terminal to respond to a CRTL-C (or perhaps the kernel killed my process for virtual memory exhaustion). That process took at least a minute. Then again, I was allocating my memory in 50MB chunks and then writing to them. Perhaps Sun's JVM on Linux allocates the whole chunk of memory at once (which doesn't do anything on Linux glibc) and then writes to the last page to see that it can be used. That doesn't seem right, so there must be some other voodoo going on here. Maybe for the sake of experimenting you could use a USB flash drive for a swap device. Even if it's just 256MB, you should be able to see if it's a swap space limitation or something more sinister... Not a bad idea, but sadly I don't have USB support available in my kernel. I have a buddy with a 128MB ppc-Linux 2.6 box with 512MB swap and java installed. I had him check, and his bombs (with a jvm panic!) when asking for even 512MB. So, it looks like physical memory and swap also have an effect. I would have expected the JVM to simply attempt to allocate the memory and use it. For instance, java -Xmx512M -Xms512M -version bombs on this little box, even though the heap is pretty much never used. The JVM must check to see if the heap request is ever going to be invalid and then simply refuses to start. Not sure how it does that quickly, though. I know that Linux will lie to a process that requests memory and indicate success when it's possible that actually using that memory will fail. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+Cso9CaO5/Lv0PARAiT/AKCytag1TNFoOkV0oE0cp3HmhrxmWwCcD1fE aKa8QlXjRcQhGCIPdV84Ngo= =eAGf -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: [OT]Re: Is better one or more Tomcat instances per machine For instance, java -Xmx512M -Xms512M -version bombs on this little box, even though the heap is pretty much never used. The JVM requests all 512 MB at startup; since your swap size is only 512 MB, it ain't gonna happen. (Swap file size - not swap plus RAM - is approximately the limit on total virtual size, since there has to be room to move pages out before the kernel can read any back in.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
| From: Christopher Schultz [mailto:[EMAIL PROTECTED] | Sent: Wednesday, 14 March, 2007 12:05 | | Perhaps, but the JVM actually refuses to start right away. In my eat | all my memory tests, I was able to eat around 1.6GB before I brought my | machine to a crawl. It took more than a minute for my terminal to | respond to a CRTL-C (or perhaps the kernel killed my process for virtual | memory exhaustion). That process took at least a minute. I imagine that when the JVM calls [cm]alloc, one of the first things alloc() does is call sbrk() to expand your process' memory space. That'll fail right away if you don't have enough VM available. If you're expanding the arena in stages, then you won't die until you hit the limit. If you want to hit the limit faster, double your allocation size each time. Once you hit the limit (malloc() should return (void *)-1), halve your allocation unit size and try again. Keep halving and allocating until you get success. Keep halving and allocating until your allocation size is less than a page. You might also want to see what results you get by just calling realloc() with increasing sizes. I may try this myself when I get home (don't have a Unix box handy here), but I'm running 64-bit Solaris on an Opteron, so I don't know if my results will help. | Then again, I was allocating my memory in 50MB chunks and then writing | to them. Perhaps Sun's JVM on Linux allocates the whole chunk of memory | at once (which doesn't do anything on Linux glibc) and then writes to | the last page to see that it can be used. That doesn't seem right, so | there must be some other voodoo going on here. What doesn't seem right? Remember, the heap isn't a bunch of empty pages; you have to set up your block list, then initialize PermGen and other system spaces. It's entirely possible that the JVM needs to access the last page as part of initializing itself. I know nothing about JVM internals and not much about Linux, but I used to be a fair VMS hacker, and I did learn a bit about virtual memory implementation (all probably obsolete if not downright wrong nowadays...) | I have a buddy with a 128MB ppc-Linux 2.6 box with 512MB swap | and java installed. I had him check, and his bombs (with a jvm panic!) | when asking for even 512MB. So, it looks like physical memory and swap | also have an effect. I would have expected the JVM to simply attempt to | allocate the memory and use it. Which is probably exactly what happened. The panic arose when the JVM attempted to initialize the heap, and found out that the kernel had lied to it about there being 512M available. Maybe the PPC kernel does some kind of optimistic allocation, where it marks the process size appropriately and just uses an allocate-on-write mechanism to permit the process to use pages up to the stated memory limit. This would permit the kernel to over-allocate VM on the pretext that processes that request large amounts of memory often don't use much of it. Kind of a VM-kiting scheme, where the kernel allocates memory it doesn't have in the hopes that when the requesting program goes to use it, some other program will have exited and freed up some space. This is all pure conjecture on my part, feel free to laugh heartily. - The information contained in this message is confidential proprietary property of Nelnet, Inc. and its affiliated companies (Nelnet) and is intended for the recipient only. Any reproduction, forwarding, or copying without the express permission of Nelnet is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this e-mail. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT]Re: Is better one or more Tomcat instances per machine
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tracy, Nelson, Tracy M. wrote: I imagine that when the JVM calls [cm]alloc, one of the first things alloc() does is call sbrk() to expand your process' memory space. That'll fail right away if you don't have enough VM available. I do not experience that scenario when allocating memory on my machine. I was able to calloc up to 3.9GB of memory on my 1GB-physical 1GB-swap machine. Linux doesn't actually allocate the memory until you try to use it, which is why I was puzzled by fast-fail behavior of the JVM. Once you hit the limit (malloc() should return (void *)-1), halve your allocation unit size and try again. This is certainly complicated by the way that glibc malloc and calloc work (by lying to the requesting program). I may try this myself when I get home (don't have a Unix box handy here), but I'm running 64-bit Solaris on an Opteron, so I don't know if my results will help. Yeah, not so much. Perhaps in a 32-bit (non-Java) virtual machine. | Then again, I was allocating my memory in 50MB chunks and then writing | to them. Perhaps Sun's JVM on Linux allocates the whole chunk of memory | at once (which doesn't do anything on Linux glibc) and then writes to | the last page to see that it can be used. That doesn't seem right, so | there must be some other voodoo going on here. What doesn't seem right? Remember, the heap isn't a bunch of empty pages; you have to set up your block list, then initialize PermGen and other system spaces. It's entirely possible that the JVM needs to access the last page as part of initializing itself. Well, I think that glibc malloc and the Linux memory manager collude to do nothing with requested-but-unused memory. So, if the JVM allocates 3GB of Java heap space and then the writes to only 5% of pages (I'm assuming that part of heap initialization is not zeroing the memory or anything stupid like that), only that small amount of memory gets allocated and the JVM should be able to start up without complaint (even though it will eventually fail due to OS trickery). Maybe the PPC kernel does some kind of optimistic allocation, where it marks the process size appropriately and just uses an allocate-on-write mechanism to permit the process to use pages up to the stated memory limit. That's exactly what glibc and Linux are documented (and observed ;) to do, which is something I've been trying to point out from the beginning. This would permit the kernel to over-allocate VM on the pretext that processes that request large amounts of memory often don't use much of it. Kind of a VM-kiting scheme, where the kernel allocates memory it doesn't have in the hopes that when the requesting program goes to use it, some other program will have exited and freed up some space. This is all pure conjecture on my part, feel free to laugh heartily. It's no joke: it happens, and when you try to use your memory that doesn't exist, the Linux kernel basically chooses a random process to kill in order to reclaim memory to allow the write to work. Usually, your own application (the JVM, in this case), just fails horribly. In my tests on the larger machine, the JVM kindly tells me that it can't give me that much memory, rather than crashing and burning as I would expect after being tricked by the OS. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF+FSf9CaO5/Lv0PARAi0uAJ96H6jR5efHLa+NhuReA/+lGeVwQwCeN5kV nZ6IQZ1iU1dikw+ocRA75o4= =wl32 -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT]Re: Is better one or more Tomcat instances per machine
| From: Christopher Schultz [mailto:[EMAIL PROTECTED] | Sent: Wednesday, 14 March, 2007 15:02 | | In my tests on the larger machine, the JVM kindly tells me that it can't | give me that much memory, rather than crashing and burning as I would | expect after being tricked by the OS. Perhaps the JVM calls getrlimit(RLIMIT_AS,...) and just takes the current soft limit as the maximum? It might be interesting to write some code to set the soft limit up to the hard limit and then exec() the JVM to see if that makes a difference. Of course, the result would probably be the crashburn, but if you're curious... I note that on later Linux kernels you can disable the overcommiting allocation behavior with: # echo 2 /proc/sys/vm/overcommit_memory - The information contained in this message is confidential proprietary property of Nelnet, Inc. and its affiliated companies (Nelnet) and is intended for the recipient only. Any reproduction, forwarding, or copying without the express permission of Nelnet is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to this e-mail. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
