RE: Setting the Right Amount of Memory
From: Mark Thomas [mailto:ma...@apache.org] Subject: Re: Setting the Right Amount of Memory So what is the impact say (picking some random numbers of the air) if you have an app that is quite happy with 256MB of heap space but has 1024MB allocated? Probably unmeasurable, assuming the larger heap has not pushed the OS into paging. I understand from what you have written that for the young gen there isn't much impact Zero impact, actually. but what about the old gen when there is a full GC? Since a full GC is a relatively rare event, and having a significant number of unreachable objects in the tenured generation is even rarer, the effect is negligible. Does the fact that there is (roughly) 700MB of additional dead objects slow down the full GC There aren't really 700 MB of dead objects when a full GC kicks in. That occurs when a survivor space is full, and there's no more room to copy objects to it from the eden space. At that point, the scan of the tenured generation space occurs, which will normally encounter only a few dead objects. (Note that it's the number of dead objects, not the size, that's of any interest here.) The eden space, where nearly all the dead objects are, does not have to be scanned for empty areas. and if so, is it significant? Probably not, unless you've got a really strange app. Doing some testing with a fairly trivial program to create lots of short-lived objects and a fixed number of cycled long-term ones shows that a larger heap improves the throughput of iterations, but does plateau after a bit. Let's see if a .java attachment makes it through... (if it doesn't and you want a look at it, let me know and I'll e-mail it directly). Sample output from running on my 2 GHz, dual-core, 4 GB laptop: C:\JavaUtiljava -Xms32m -Xmx32m TestGC 1 total objects: 1, objects to keep: 1000, keep interval: 1000, object size: 1000 14:07:56.203 - 14:08:58.869 1595761.657 objects per second C:\JavaUtiljava -Xms32m -Xmx32m TestGC 1 total objects: 1, objects to keep: 1000, keep interval: 1000, object size: 1000 14:09:06.209 - 14:10:09.995 1567742.137 objects per second C:\JavaUtiljava -Xms512m -Xmx512m TestGC 1 total objects: 1, objects to keep: 1000, keep interval: 1000, object size: 1000 14:10:27.605 - 14:11:25.914 1715001.114 objects per second C:\JavaUtiljava -Xms512m -Xmx512m TestGC 1 total objects: 1, objects to keep: 1000, keep interval: 1000, object size: 1000 14:11:35.517 - 14:12:33.195 1733763.306 objects per second C:\JavaUtiljava -Xms2g -Xmx2g TestGC 1 total objects: 1, objects to keep: 1000, keep interval: 1000, object size: 1000 14:12:57.895 - 14:13:55.462 1737106.328 objects per second C:\JavaUtiljava -Xms2g -Xmx2g TestGC 1 total objects: 1, objects to keep: 1000, keep interval: 1000, object size: 1000 14:14:01.601 - 14:14:58.758 1749566.982 objects per second - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
This email contained a .zip file attachment. Raytheon does not allow email attachments that are considered likely to contain malicious code. For your protection this attachment has been removed. If this email is from an unknown source, please simply delete this email. If this email was expected, and it is from a known sender, you may follow the below suggested instructions to obtain these types of attachments. + Instruct the sender to enclose the file(s) in a .zip compressed file, and rename the .zip compressed file with a different extension, such as, .rtnzip. Password protecting the renamed .zip compressed file adds an additional layer of protection. When you receive the file, please rename it with the extension .zip. Additional instructions and options on how to receive these attachments can be found at: http://security.it.ray.com/antivirus/extensions.html http://security.it.ray.com/news/2007/zipfiles.html Should you have any questions or difficulty with these instructions, please contact the Help Desk at 877.844.4712 --- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Subject: RE: Setting the Right Amount of Memory Let's see if a .java attachment makes it through... So let's try it as a zip file... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
On 21/06/2010 04:15, Caldarale, Charles R wrote: From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Subject: Re: Setting the Right Amount of Memory Unreachable (dead) objects are never encountered, so their number or size does not come into play. For complete disclosure, I should note that dead objects in the tenured (old) and permanent generations are looked at (but only during a full GC), since the compaction phase has to figure out where the live ones can be copied to. However, since in nearly all cases, more than 90% of dead objects are in the young generation, few dead objects are examined and the cost is minimal. The compaction phase for the young generation (minor GC) actually consists of copying the live objects to a survivor space, which is guaranteed to be empty. So what is the impact say (picking some random numbers of the air) if you have an app that is quite happy with 256MB of heap space but has 1024MB allocated? I understand from what you have written that for the young gen there isn't much impact but what about the old gen when there is a full GC? Does the fact that there is (roughly) 700MB of additional dead objects slow down the full GC and if so, is it significant? It could easily represent an order of magnitude increase is the memory used by dead objects. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
So what is the impact say (picking some random numbers of the air) if you have an app that is quite happy with 256MB of heap space but has 1024MB allocated? My question exactly! -- Eric Robinson Disclaimer - June 21, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
From: Robinson, Eric [mailto:eric.robin...@psmnv.com] Subject: RE: Setting the Right Amount of Memory So what is the impact say (picking some random numbers of the air) if you have an app that is quite happy with 256MB of heap space but has 1024MB allocated? My question exactly! Sorry for not responding yet. Busy at what I actually get paid for, and trying to generate some real numbers for the question at hand. More later. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
Sorry for not responding yet. Busy at what I actually get paid for, and trying to generate some real numbers for the question at hand. More later. No worries, I just want to keep the thread alive until something definitive comes from it if possible. :-) -- Eric Robinson Disclaimer - June 21, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
Caldarale, Charles R wrote: On Jun 19, 2010, at 18:31, André Warnier a...@ice-sa.com wrote: As Mark writes above (and my interpretation of things) : - a bigger Heap means that the JVM will be able to accumulate more dead stuff in it, - when it is needed however, it will take much longer, because there is more stuff to clean up. I thought we had taught you better than that... Ooops. The time it takes to perform a GC is *not* dependent on the number or size of dead objects, just on the live ones. That is counter-intuitive. Pray, why is that ? (Or point again to a page describing the process, maybe. That would be useful for the OP also). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[OT] Re: Setting the Right Amount of Memory
Caldarale, Charles R wrote: (Sent from my iPhone on a ferry in the middle of Lake Michigan.) Posters to this forum, observe the incredible dedication of some of the contributors here. I'm willing to bet that if the ferry was sinking, Chuck would be the last one on board, making sure there wasn't any unanswered message on this forum (or at least any wrong and uncorrected answer lingering). We should have a competition about whom can post a message from the most unlikely location, or circumstances. The middle of lake Michigan isn't bad for a start. We would need some means of checking though. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: Setting the Right Amount of Memory
On 20/06/2010 10:25, André Warnier wrote: Caldarale, Charles R wrote: We would need some means of checking though. Some means of registering a check in, by location? Hmm. I wonder if anyone's had that idea before... ;) p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org signature.asc Description: OpenPGP digital signature
Re: Setting the Right Amount of Memory
On 20/06/2010 00:30, André Warnier wrote: Robinson, Eric wrote: On 17/06/2010 08:59, Robinson, Eric wrote: If your heap size is right on the edge of your minimum for a Tomcat instance, you may be doing more GC work than is really needed. However, if you're satisfied with the response time and CPU utilization, you should be ok. Time to hit the vendor around the head with the cluebat. If the app is happy with less heap space then increasing it is only going to cause problems - mainly that GC when it happens will take longer and trigger longer pauses. You can mitigate this with GC config (later VMs may make the right choices for you anyway) but all this is just adding unecessary complexity. Mark With 160 instances of tomcat on the server, and most of them happy with 64-96MB of RAM, could you take an educated guess at the negative impact on the server of raising the RAM to 512MB for each instance? How much extra CPU utilization do you think I could possibly see from all the extra GC? Just a note here : 160 X 512 MB = 81 GB. If each Tomcat's JVM is allowed to use up to 512 MB of Heap, there might be moments where a lot of JVM's will be using close to that amount. Unless your system can really support that amount of real RAM, you may be in for massive swapping. And that could be a major problem. JVMs and swapping do not place nicely together. To quote some folks at work that have been looking at this recently performance falls of a cliff. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
On 20/06/2010 10:24, André Warnier wrote: Caldarale, Charles R wrote: On Jun 19, 2010, at 18:31, André Warnier a...@ice-sa.com wrote: As Mark writes above (and my interpretation of things) : - a bigger Heap means that the JVM will be able to accumulate more dead stuff in it, - when it is needed however, it will take much longer, because there is more stuff to clean up. I thought we had taught you better than that... Ooops. Me too. I was under the impression more dead objects == longer GC. The time it takes to perform a GC is *not* dependent on the number or size of dead objects, just on the live ones. That is counter-intuitive. Pray, why is that ? +1. I'd to get me head around this. (Or point again to a page describing the process, maybe. That would be useful for the OP also). Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
On Jun 20, 2010, at 4:24, André Warnier a...@ice-sa.com wrote: The time it takes to perform a GC is *not* dependent on the number or size of dead objects, just on the live ones. That is counter-intuitive. Pray, why is that ? All modern GC algorithms are variations of mark-sweep-compact. The basic operation consists of following the object reference graph from a set of known roots (eg, thread stack frames), marking each discovered object with a found flag, and copying the found objects to an unoccupied area. Unreachable (dead) objects are never encountered, so their number or size does not come into play. The space occupied by dead objects is automatically reclaimed as live objects are copied over them. There are links to various papers and descriptions on the Java technologies page: http://java.sun.com/javase/technologies/hotspot/gc-index.jsp Googling for mark-sweep-compact will get you some of the academic papers and continuing research into the topic. - Chuck (Now in a hotel watching the All Whites vs the Azzurri.)
Re: Setting the Right Amount of Memory
On 20.6.2010 14:06, Mark Thomas wrote: On 20/06/2010 00:30, André Warnier wrote: Just a note here : 160 X 512 MB = 81 GB. If each Tomcat's JVM is allowed to use up to 512 MB of Heap, there might be moments where a lot of JVM's will be using close to that amount. Unless your system can really support that amount of real RAM, you may be in for massive swapping. And that could be a major problem. JVMs and swapping do not place nicely together. To quote some folks at work that have been looking at this recently performance falls of a cliff. Yep, I've seen that as well -- and the effect really is easy to understand, esp. in the light of how Chuck explained the mark-sweep-compact memory management. The live objects are somewhat scattered throughout the heap, and walking through the live object graph will touch all of them -- thus if any was paged out at the moment, it will be paged in (most likely forcing some other page out). If majority of the OS-level virtual memory consumption consists of JVM heaps, then it's very likely that the paged-out memory page does contain a live Java object - which will again be paged in when the owning JVM does its mark phase. Paging is suitable OS-level memory management for processes with full pages of data that only needs to be accessed infrequently (or perhaps only in the initialization phase of the application). Then these seldom-accessed pages can be moved out to disk, freeing RAM for more frequently needed items. With JVM memory management, everything is accessed every now and then, making paging a real performance-killer. -- ..Juha - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
On 20 June 2010 16:01, Caldarale, Charles R chuck.caldar...@unisys.comwrote: All modern GC algorithms are variations of mark-sweep-compact. The basic operation consists of following the object reference graph from a set of known roots (eg, thread stack frames), marking each discovered object with a found flag, and copying the found objects to an unoccupied area. Unreachable (dead) objects are never encountered, so their number or size does not come into play. The space occupied by dead objects is automatically reclaimed as live objects are copied over them. Note that in some variants, the cost of a compact *does* vary with the number of dead objects. In Squeak (Smalltalk), for example, each object header contains the object size, and hence by implication the address of the next object header. Squeak's GC compacts (after the mark phase) by starting at the bottom of object memory and iterating through each object, copying down the live ones and updating pointers on the way. In Java, this would only ever be necessary in OldSpace - with a generational GC where you're collecting anything other than the oldest generation, you just copy the live objects out as you encounter them and update any pointers to other objects in the same generation, after which you know the space is empty. Typically, the speed of such a compact is limited by the speed of access to memory. Most modern memory architectures will treat such sequential read access specially and start to read-ahead, so the memory overhead isn't as bad as it might be. My GC knowledge is a little out of date, so if any of the current GC experts wish to correct me I'll be interested! - Peter
RE: Setting the Right Amount of Memory
Having a borderline heap size can, in the worst case, result in almost continual GC activity, if there is only room to allocate a minimal number of objects between GC occurrences. For what it's worth, either this is not the case in our real-world situation or the effect is negligible. Even though we have 160 instances of tomcat on the same server, and all of them are trimmed to use the minimum RAM (usually 64-96MB) and all instances are quite actively being used, the server is still 90% idle during peak load. The server is an 8-core 2.6GHz Xeon with 32GB RAM (10GB free and no swapping). Disclaimer - June 20, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Subject: Re: Setting the Right Amount of Memory Unreachable (dead) objects are never encountered, so their number or size does not come into play. For complete disclosure, I should note that dead objects in the tenured (old) and permanent generations are looked at (but only during a full GC), since the compaction phase has to figure out where the live ones can be copied to. However, since in nearly all cases, more than 90% of dead objects are in the young generation, few dead objects are examined and the cost is minimal. The compaction phase for the young generation (minor GC) actually consists of copying the live objects to a survivor space, which is guaranteed to be empty. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: Setting the Right Amount of Memory
From: Robinson, Eric [mailto:eric.robin...@psmnv.com] Subject: RE: Setting the Right Amount of Memory Having a borderline heap size can, in the worst case, result in almost continual GC activity, if there is only room to allocate a minimal number of objects between GC occurrences. For what it's worth, either this is not the case in our real-world situation or the effect is negligible. Not surprising - you'd have to be very unlucky to be right at the edge and see a lot of GC activity and be able to continue running. Usually you'll be over the edge a bit, encounter OOMEs, increase the heap significantly, and be well away from the frequent tight heap scenario. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
For what it's worth, either this is not the case in our real-world situation or the effect is negligible. Not surprising - you'd have to be very unlucky to be right at the edge and see a lot of GC activity and be able to continue running. Usually you'll be over the edge a bit, encounter OOMEs, increase the heap significantly, and be well away from the frequent tight heap scenario. What qualifies as a tight heap and what qualifies as a significant increase? Usually when we see OOMEs we increase the allocation by 32MB and they go away. To return to the original question, is it generally better to custom-fit the RAM allocation (as we currently do) or to unilaterally set all instances to a higher amount that we know will not generate OOMEs, such as 512MB? -- Eric Robinson Disclaimer - June 20, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
From: Robinson, Eric [mailto:eric.robin...@psmnv.com] Subject: RE: Setting the Right Amount of Memory What qualifies as a tight heap and what qualifies as a significant increase? Both are entirely dependent on what's running inside the JVM. Monitoring the GC actions will tell you if you're close to being tight. Usually when we see OOMEs we increase the allocation by 32MB and they go away. From your descriptions, none of your webapps are memory-intensive, so 32 MB is likely a significant increase in the heap size - in your case. is it generally better to custom-fit the RAM allocation (as we currently do) or to unilaterally set all instances to a higher amount that we know will not generate OOMEs, such as 512MB? Depends on how much time you've got to spend on the problem, and how much RAM you've got. If the RAM is available, it's certainly easier to set the heap size large for everyone and let it rip. If you are constrained such that doing so will result in an overcommitment of RAM (or you like things to be just right), then you need to individually tune. You might actually have a situation where it would work to set -Xms small (16 or 32 MB) and -Xmx large (512 MB), and let each JVM figure out what it really needs. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
On 17/06/2010 08:59, Robinson, Eric wrote: If your heap size is right on the edge of your minimum for a Tomcat instance, you may be doing more GC work than is really needed. However, if you're satisfied with the response time and CPU utilization, you should be ok. Time to hit the vendor around the head with the cluebat. If the app is happy with less heap space then increasing it is only going to cause problems - mainly that GC when it happens will take longer and trigger longer pauses. You can mitigate this with GC config (later VMs may make the right choices for you anyway) but all this is just adding unecessary complexity. Mark With 160 instances of tomcat on the server, and most of them happy with 64-96MB of RAM, could you take an educated guess at the negative impact on the server of raising the RAM to 512MB for each instance? How much extra CPU utilization do you think I could possibly see from all the extra GC? --Eric Disclaimer - June 19, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
Robinson, Eric wrote: On 17/06/2010 08:59, Robinson, Eric wrote: If your heap size is right on the edge of your minimum for a Tomcat instance, you may be doing more GC work than is really needed. However, if you're satisfied with the response time and CPU utilization, you should be ok. Time to hit the vendor around the head with the cluebat. If the app is happy with less heap space then increasing it is only going to cause problems - mainly that GC when it happens will take longer and trigger longer pauses. You can mitigate this with GC config (later VMs may make the right choices for you anyway) but all this is just adding unecessary complexity. Mark With 160 instances of tomcat on the server, and most of them happy with 64-96MB of RAM, could you take an educated guess at the negative impact on the server of raising the RAM to 512MB for each instance? How much extra CPU utilization do you think I could possibly see from all the extra GC? Just a note here : 160 X 512 MB = 81 GB. If each Tomcat's JVM is allowed to use up to 512 MB of Heap, there might be moments where a lot of JVM's will be using close to that amount. Unless your system can really support that amount of real RAM, you may be in for massive swapping. As Mark writes above (and my interpretation of things) : - a bigger Heap means that the JVM will be able to accumulate more dead stuff in it, before needing a full GC to clean it up, than if the Heap was smaller. - thus, all else being equal, it will take more time elapsed before a full-GC is needed. - when it is needed however, it will take much longer, because there is more stuff to clean up. So all in all, you will have less GC's, but each one will take longer. During a full GC, the JVM does nothing else (applications are frozen). So there may be an impact on the user experience during that time. But, if a significant number of JVMs start filling up their Heap simultaneously, then the memory used will be real, not just reserved. And if your system has less physical RAM than the total required at any one time, it starts swapping. That /would/ have a definite impact on the user experience. Note also that the 81 GB above count only the Tomcat Heaps, and disregard any other memory needed by your 160 instances. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
Just a note here : 160 X 512 MB = 81 GB. If each Tomcat's JVM Is allowed to use up to 512 MB of Heap, there might be moments where a lot of JVM's will be using close to that amount. Unless your system can really support that amount of real RAM, you may be in for massive swapping. Thanks, but no worries there. I would never allow swapping. If I have to allocate 512MB RAM to each instance, it will be on a new server with adequate physical RAM. During a full GC, the JVM does nothing else (applications are frozen). So there may be an impact on the user experience during that time. That's the effect I'm trying to anticipate. Note also that the 81 GB above count only the Tomcat Heaps, and disregard any other memory needed by your 160 instances. Very good point. -- Eric Robinson Disclaimer - June 19, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
On Jun 19, 2010, at 18:31, André Warnier a...@ice-sa.com wrote: As Mark writes above (and my interpretation of things) : - a bigger Heap means that the JVM will be able to accumulate more dead stuff in it, - when it is needed however, it will take much longer, because there is more stuff to clean up. I thought we had taught you better than that... The time it takes to perform a GC is *not* dependent on the number or size of dead objects, just on the live ones. Having a borderline heap size can, in the worst case, result in almost continual GC activity, if there is only room to allocate a minimal number of objects between GC occurrences. - Chuck (Sent from my iPhone on a ferry in the middle of Lake Michigan.)
RE: Setting the Right Amount of Memory
If your heap size is right on the edge of your minimum for a Tomcat instance, you may be doing more GC work than is really needed. However, if you're satisfied with the response time and CPU utilization, you should be ok. My thoughts exactly. Just wanted to check it with the community. FYI, with 160 instances of tomcat running at the same time, CPU is still 90% idle even during peak production hours. Now the software vendor is coming along and forcing us to set the heap for each instance to 512MB when 64MB or 96MB works fine. It's unnecessarily expensive and super frustrating. -- Eric Robinson Disclaimer - June 17, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
Just wondering, what tools do you use to manage all the instances? Also, what do you use to look for the OutOfMemory in logs? I am looking at Splunk too. Just shell scripts. We have scripts that... -- set up a new instance and customize it for a new customer -- stops, starts, restarts, optionally clearing the work folders -- analyzes jasper logs for system responsiveness (both realtime and historically) -- synchronizes certain files between instances on different tomcat servers -- watches for OutOfMemory in various logs -- grooms logs periodically -- monitors various logs in realtime ..and so on. Never found anything we couldn't do with bash and cron. -- Eric Robinson Disclaimer - June 17, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
On 17/06/2010 08:59, Robinson, Eric wrote: If your heap size is right on the edge of your minimum for a Tomcat instance, you may be doing more GC work than is really needed. However, if you're satisfied with the response time and CPU utilization, you should be ok. My thoughts exactly. Just wanted to check it with the community. FYI, with 160 instances of tomcat running at the same time, CPU is still 90% idle even during peak production hours. Now the software vendor is coming along and forcing us to set the heap for each instance to 512MB when 64MB or 96MB works fine. It's unnecessarily expensive and super frustrating. Time to hit the vendor around the head with the cluebat. If the app is happy with less heap space then increasing it is only going to cause problems - mainly that GC when it happens will take longer and trigger longer pauses. You can mitigate this with GC config (later VMs may make the right choices for you anyway) but all this is just adding unecessary complexity. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
Time to hit the vendor around the head with the cluebat. If the app is happy with less heap space then increasing it is only going to cause problems - mainly that GC when it happens will take longer and trigger longer pauses. You can mitigate this with GC config (later VMs may make the right choices for you anyway) but all this is just adding unnecessary complexity. I agree completely. I wish the vendor accepted feedback like this. Unfortunately if you try to show them a better way, they tend to get defensive and think of you as a troublemaker. :-( -- Eric Robinson Disclaimer - June 17, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
On 6/16/2010 10:42 AM, Robinson, Eric wrote: We run 150+ instances of tomcat on one server. To optimize memory utilization, each tomcat instance is configured with 64MB by default (export JAVA_OPTS=-ms64M -mx64M). We then watch for Java.Lang.OutOfmemory errors in the logs. If we see any of these over time, we allocate additional RAM in 32MB increments until that instance stops throwing errors. Most instances are happy with 64MB. Some need 96MB or 128MB. We have never seen a tomcat instance that throws OutOfMemory errors with 256MB RAM. If tomcat is not throwing OutOfMemory errors and system responsiveness is good, is there really any reason to allocate higher amounts of RAM? Nope. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Setting the Right Amount of Memory
Hi Eric, Just wondering, what tools do you use to manage all the instances? Also, what do you use to look for the OutOfMemory in logs? I am looking at Splunk too. Andrew On Thu, Jun 17, 2010 at 12:42 AM, Robinson, Eric eric.robin...@psmnv.com wrote: We run 150+ instances of tomcat on one server. To optimize memory utilization, each tomcat instance is configured with 64MB by default (export JAVA_OPTS=-ms64M -mx64M). We then watch for Java.Lang.OutOfmemory errors in the logs. If we see any of these over time, we allocate additional RAM in 32MB increments until that instance stops throwing errors. Most instances are happy with 64MB. Some need 96MB or 128MB. We have never seen a tomcat instance that throws OutOfMemory errors with 256MB RAM. If tomcat is not throwing OutOfMemory errors and system responsiveness is good, is there really any reason to allocate higher amounts of RAM? -- Eric Robinson Disclaimer - June 16, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Setting the Right Amount of Memory
From: Robinson, Eric [mailto:eric.robin...@psmnv.com] Subject: Setting the Right Amount of Memory If tomcat is not throwing OutOfMemory errors and system responsiveness is good, is there really any reason to allocate higher amounts of RAM? If your heap size is right on the edge of your minimum for a Tomcat instance, you may be doing more GC work than is really needed. However, if you're satisfied with the response time and CPU utilization, you should be ok. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
