Re: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
On 28/12/17 10:06, Olaf Kock wrote: > > On 27.12.2017 23:16, Eric Robinson wrote: >>> I mean A is java8 and tomcat8.. so make a C that is tomcat6 and java8 >> I don't think so. This is a requirement of the software company whose >> application solution we use. They are requiring us to move to tomcat 8 >> with jdk 1.8. If we try to mix tomcat8 with jdk 1.6, supposedly we >> would have problems. I guess all this is being driven by the need to >> switch to TLS 1.2. I'm not sure if that would be a function of tomcat >> or java. > As you were looking for the reason of the increased footprint, this > would help you tremendously to get to the root cause, even if you don't > intend to use it in production. Assume that a similar behavior already > appears when you run tomcat6 with Java8 - in this case there might be > little reason to continue to dig in tomcat, and assume that it's the > Java implementation that caused your increased footprint. > > Tomcat 8 with Java 6 won't work (apart from the outdated implementation) > as it requires at least Java7 (which is also outdated). > > And then there's yet another aspect: The memory footprint increased way > below the price decrease for memory, so just adding this much memory > could be filed away as a reasonable assumption within 7.5 years (JDK > 1.6.21 was released July 2010, I'm assuming that this is the age of the > hardware as well (because why would you have installed this version on > newer hardware, when newer releases existed) > > I'm assuming that you have enough aspects to inspect - if you're really > interested in finding the root cause, you'll need to come up with more > specific measurements, e.g. profiler data, compare thread dumps and set > up Tomcat 6 with Java 8 to have a third reference point. I don't have Tomcat 6 to hand but I do have Tomcat 7. A quick test with Tomcat 7 + Java 6 vs Tomcat 7 + Java 8 I saw a 20% increase in memory usage. This looks very much like changes in the JVM rather than changes in Tomcat. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
On Thu, Dec 28, 2017 at 3:36 PM, Olaf Kock wrote: > > On 27.12.2017 23:16, Eric Robinson wrote: >>> >>> I mean A is java8 and tomcat8.. so make a C that is tomcat6 and java8 >> >> I don't think so. This is a requirement of the software company whose >> application solution we use. They are requiring us to move to tomcat 8 with >> jdk 1.8. If we try to mix tomcat8 with jdk 1.6, supposedly we would have >> problems. I guess all this is being driven by the need to switch to TLS 1.2. >> I'm not sure if that would be a function of tomcat or java. > > As you were looking for the reason of the increased footprint, this would > help you tremendously to get to the root cause, even if you don't intend to > use it in production. Assume that a similar behavior already appears when > you run tomcat6 with Java8 - in this case there might be little reason to > continue to dig in tomcat, and assume that it's the Java implementation that > caused your increased footprint. > > Tomcat 8 with Java 6 won't work (apart from the outdated implementation) as > it requires at least Java7 (which is also outdated). > > And then there's yet another aspect: The memory footprint increased way > below the price decrease for memory, so just adding this much memory could > be filed away as a reasonable assumption within 7.5 years (JDK 1.6.21 was > released July 2010, I'm assuming that this is the age of the hardware as > well (because why would you have installed this version on newer hardware, > when newer releases existed) > > I'm assuming that you have enough aspects to inspect - if you're really > interested in finding the root cause, you'll need to come up with more > specific measurements, e.g. profiler data, compare thread dumps and set up > Tomcat 6 with Java 8 to have a third reference point. > > Olaf > How about turning on Native Memory Tracking for both Tomcat8+JDK1.8 and Tomcat6+JDK1.6 combination and compare the results? It will give you details about internal memory consumption of the JVM. If you see increase there, you will also able to find out which component's memory footprint has increased. You can turn NMT on by adding following JVM arguments: -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary After turning on NMT and running Tomcat, when you see there is rise in memory usage, use JCMD to dump native memory consumption details for analysis. You can find more details about NMT here: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html IMO, if you see there is not much difference between native memory consumption of JDK1.6 and JDK1.8, then you need to focus on Tomcat. Thanks! Suvendu - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
On 27.12.2017 23:16, Eric Robinson wrote: I mean A is java8 and tomcat8.. so make a C that is tomcat6 and java8 I don't think so. This is a requirement of the software company whose application solution we use. They are requiring us to move to tomcat 8 with jdk 1.8. If we try to mix tomcat8 with jdk 1.6, supposedly we would have problems. I guess all this is being driven by the need to switch to TLS 1.2. I'm not sure if that would be a function of tomcat or java. As you were looking for the reason of the increased footprint, this would help you tremendously to get to the root cause, even if you don't intend to use it in production. Assume that a similar behavior already appears when you run tomcat6 with Java8 - in this case there might be little reason to continue to dig in tomcat, and assume that it's the Java implementation that caused your increased footprint. Tomcat 8 with Java 6 won't work (apart from the outdated implementation) as it requires at least Java7 (which is also outdated). And then there's yet another aspect: The memory footprint increased way below the price decrease for memory, so just adding this much memory could be filed away as a reasonable assumption within 7.5 years (JDK 1.6.21 was released July 2010, I'm assuming that this is the age of the hardware as well (because why would you have installed this version on newer hardware, when newer releases existed) I'm assuming that you have enough aspects to inspect - if you're really interested in finding the root cause, you'll need to come up with more specific measurements, e.g. profiler data, compare thread dumps and set up Tomcat 6 with Java 8 to have a third reference point. Olaf - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
> > More heap or more native memory? > > > > With the exact same Xms and Xmx settings, I get vastly different resident and > virtual image sizes from the Linux ps command. > > > tomcatA: jdk1.8.0_152, res: 694312, virt: 5045084 > tomcatB: jdk1.6.0_21, res: 332840, virt: 3922656 > > > And b is also tomcat8 right? No, tomcatB is using tomcat6. > Can you make that also tomcat8 but keep java8? > > > I mean A is java8 and tomcat8.. so make a C that is tomcat6 and java8 > > I don't think so. This is a requirement of the software company whose application solution we use. They are requiring us to move to tomcat 8 with jdk 1.8. If we try to mix tomcat8 with jdk 1.6, supposedly we would have problems. I guess all this is being driven by the need to switch to TLS 1.2. I'm not sure if that would be a function of tomcat or java. --Eric
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
Op 23 dec. 2017 09:27 schreef "Johan Compagner" : Op 22 dec. 2017 21:02 schreef "Eric Robinson" : > > More heap or more native memory? > With the exact same Xms and Xmx settings, I get vastly different resident and virtual image sizes from the Linux ps command. tomcatA: jdk1.8.0_152, res: 694312, virt: 5045084 tomcatB: jdk1.6.0_21, res: 332840, virt: 3922656 And b is also tomcat8 right? Can you make that also tomcat8 but keep java8? I mean A is java8 and tomcat8.. so make a C that is tomcat6 and java8 --Eric - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
Op 22 dec. 2017 21:02 schreef "Eric Robinson" : > > More heap or more native memory? > With the exact same Xms and Xmx settings, I get vastly different resident and virtual image sizes from the Linux ps command. tomcatA: jdk1.8.0_152, res: 694312, virt: 5045084 tomcatB: jdk1.6.0_21, res: 332840, virt: 3922656 And b is also tomcat8 right? Can you make that also tomcat8 but keep java8? --Eric - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
On 22.12.2017 21:02, Eric Robinson wrote: With the exact same Xms and Xmx settings, I get vastly different resident and virtual image sizes from the Linux ps command. tomcatA: jdk1.8.0_152, res: 694312, virt: 5045084 tomcatB: jdk1.6.0_21, res: 332840, virt: 3922656 -Xmx is not all that's determining how much memory the JVM actually allocates. https://jguru.fi/why-is-my-java-process-taking-more-memory-than-i-gave-it.html gives some more hints on factors that have to be taken into account. 32 vs 64 bit architectures might do something to the sizes. And, now that you gave your JVM options in another answer, you're not specifying the GC algorithm and parameters, other than just logging. This means that most likely you're using another algorithm with different parameters, e.g. it might kick in later. Coming back to the linked article: Tomcat might have different default thread pool sizes - I don't know if you explicitly configure them. And I've lost track if the default connectors are different ones between Tomcat 6 and 8, or if you have ex- or implicitly configured them differently (e.g. through using your Linux distribution's implementation and they might have changed it). Unrelated: I like to configure my production servers with identical -Xms and -Xmx, so that they either start or don't start when I'm around (or when the server boots) and not fail to allocate more memory from the OS sunday night at 3am. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
> Eric, > > Just curious how much ram do you have in the server and cpu resources. > > #free -m and # cat /proc/cpuinfo | egrep 'cores|processor' > > (Not to insult your intelligence , I am just specifying what I was curious to > see) > > And it's always easier to copy/paste than to think. > > I see in another thread you went from Java 1.6_xxx to 1.8_xxx > > That could be the whole story right there. > > No offense taken. You're right, copy and paste is easier... [root@app17 alley]# free -m total used free sharedbuffers cached Mem: 64415 58110 6304 0 2938 18382 -/+ buffers/cache: 36789 27626 Swap:15999759 15240 [root@app17 alley]# cat /proc/cpuinfo | egrep 'cores|processor' processor : 0 cpu cores : 6 processor : 1 cpu cores : 6 processor : 2 cpu cores : 6 processor : 3 cpu cores : 6 processor : 4 cpu cores : 6 processor : 5 cpu cores : 6 processor : 6 cpu cores : 6 processor : 7 cpu cores : 6 processor : 8 cpu cores : 6 processor : 9 cpu cores : 6 processor : 10 cpu cores : 6 processor : 11 cpu cores : 6 --Eric - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
Eric, Just curious how much ram do you have in the server and cpu resources. #free -m and # cat /proc/cpuinfo | egrep 'cores|processor' (Not to insult your intelligence , I am just specifying what I was curious to see) And it's always easier to copy/paste than to think. I see in another thread you went from Java 1.6_xxx to 1.8_xxx That could be the whole story right there. === Thank You; Chris Cheltenham Technology Services The School District of Philadelphia Work # 215-400-5025 Cell # 215-301-6571 -Original Message- From: Eric Robinson [mailto:eric.robin...@psmnv.com] Sent: Friday, December 22, 2017 2:59 PM To: Tomcat Users List Subject: RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6? > > From: Eric Robinson [mailto:eric.robin...@psmnv.com] > > Subject: RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory > > Than > Tomcat 6? > > > if JVM instance "A" is configured as follows on server 1 under > > tomcat6/jdk > 1.6, then > > instance "A" on server 2 is configured the same, except it is > > running > under tomcat > > 8/jdk 1.8. Yet the tomcat 8 ones used 50% more memory, on average. > > > JAVA_OPTS="-Xms16M -Xmx192M \ > > -XX:MaxPermSize=192M \ > > Note that PermGen is no longer used in Java 8, and the above should > log a > warning: > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > MaxPermSize=192m; support was removed in 8.0 Great tip, thanks. Unfortunately, I don't think that explains why the exact same Xms and Xmx settings produce vastly different resident and virtual running image sizes under jdk1.8 versus jdk1.6. > It might be leaking. If that were the case, I assume it would manifest under tomcat6/jdk 1.6 as well. Since it does not, I am inclined to think leakage is not the issue. --Eric - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
> > More heap or more native memory? > With the exact same Xms and Xmx settings, I get vastly different resident and virtual image sizes from the Linux ps command. tomcatA: jdk1.8.0_152, res: 694312, virt: 5045084 tomcatB: jdk1.6.0_21, res: 332840, virt: 3922656 --Eric - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
> > From: Eric Robinson [mailto:eric.robin...@psmnv.com] > > Subject: RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than > Tomcat 6? > > > if JVM instance "A" is configured as follows on server 1 under tomcat6/jdk > 1.6, then > > instance "A" on server 2 is configured the same, except it is running > under tomcat > > 8/jdk 1.8. Yet the tomcat 8 ones used 50% more memory, on average. > > > JAVA_OPTS="-Xms16M -Xmx192M \ > > -XX:MaxPermSize=192M \ > > Note that PermGen is no longer used in Java 8, and the above should log a > warning: > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > MaxPermSize=192m; > support was removed in 8.0 Great tip, thanks. Unfortunately, I don't think that explains why the exact same Xms and Xmx settings produce vastly different resident and virtual running image sizes under jdk1.8 versus jdk1.6. > It might be leaking. If that were the case, I assume it would manifest under tomcat6/jdk 1.6 as well. Since it does not, I am inclined to think leakage is not the issue. --Eric - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
> From: Eric Robinson [mailto:eric.robin...@psmnv.com] > Subject: RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6? > if JVM instance "A" is configured as follows on server 1 under tomcat6/jdk 1.6, then > instance "A" on server 2 is configured the same, except it is running under tomcat > 8/jdk 1.8. Yet the tomcat 8 ones used 50% more memory, on average. > JAVA_OPTS="-Xms16M -Xmx192M \ > -XX:MaxPermSize=192M \ Note that PermGen is no longer used in Java 8, and the above should log a warning: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=192m; support was removed in 8.0 Here's a brief overview: https://blogs.oracle.com/poonam/about-g1-garbage-collector,-permanent-genera tion-and-metaspace - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. smime.p7s Description: S/MIME cryptographic signature
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
Eric, If you have upgraded java along with tomcat then yes that it is very probable. You can restrict how much memory java can use however, if it is consuming too much memory. -Xmx and –Xms startup parameters. However, you may be jeopardizing performance. In this case you can only add more memory. It may also be leaking. Java is a pig get used to it. === Thank You; Chris Cheltenham Technology Services The School District of Philadelphia Work # 215-400-5025 Cell # 215-301-6571 From: Eric Robinson [mailto:eric.robin...@psmnv.com] Sent: Friday, December 22, 2017 2:04 PM To: Tomcat Users List Subject: RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
RE: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
> On 22.12.2017 13:48, Eric Robinson wrote: > > We have multiple JVMs deployed on two identical Linux servers. Each server > has 60 JVMs. Until today, both servers were running Tomcat6 with JDK 1.6. > Today we upgraded one of the servers to Tomcat 8 with JDK 1.8. Now the JVMs > on the Tomcat 8 server are each using between 20-80% more memory than the > ones on Tomcat6 with JDK 1.6. Is that normal? Why would that be? Is it some > kind of settings? Is it fixable? > > > It might be as simple as different thresholds for the garbage collector to > kick in. > I'd start with an evaluation of how much memory is used right after a GC run - > and in case this isn't satisfactory, which objects use the memory. > Typically it's appropriate to just look at the top of the list. > > Note that the GC algorithms (or just GC defaults) between the different JVM > versions (sometimes even between minor upgrades) might differ significantly. > One big question is: Did you explicitly configure memory consumption, GC > algorithm and thresholds? If so, what's the difference between the two > options: > I'd expect that you need to change the settings significantly in order to > achieve > the same behavior. There's a lot of work that has been done in this world. > > That being said, I'd also not rule out that tomcat's or other component's > implementation changed - e.g. caches, or just memory use through upgraded > libraries. But I'd recommend to look in both directions, with JDK and GC > tuning > being the elephant in the room, giving you the biggest bang for your buck. > > Olaf > The following startup options are typical for our JVMs. The min, max, and permgen settings may differ from JVM to JVM on the same server, but the settings are always identical between servers. In other words, if JVM instance "A" is configured as follows on server 1 under tomcat6/jdk 1.6, then instance "A" on server 2 is configured the same, except it is running under tomcat 8/jdk 1.8. Yet the tomcat 8 ones used 50% more memory, on average. JAVA_OPTS="-Xms16M -Xmx192M \ -XX:MaxPermSize=192M \ -Djvm=$JVM_ID \ -Djava.awt.headless=true \ -Djava.net.preferIPv4Stack=true \ -Duser.timezone=US/Pacific \ -Xloggc:/alley/site098/tomcat8/logs/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails
Re: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Eric, On 12/22/17 7:48 AM, Eric Robinson wrote: > We have multiple JVMs deployed on two identical Linux servers. > Each server has 60 JVMs. Until today, both servers were running > Tomcat6 with JDK 1.6. Today we upgraded one of the servers to > Tomcat 8 with JDK 1.8. Now the JVMs on the Tomcat 8 server are each > using between 20-80% more memory than the ones on Tomcat6 with JDK > 1.6. Is that normal? Why would that be? Is it some kind of > settings? Is it fixable? More heap or more native memory? - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlo9STQdHGNocmlzQGNo cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFgXQhAAnOCilGyuELDsZq+r r25nW+9spA6ilZqB5fLMNJkBNj/93AB9tzVbtwZqMKPvPo2KD78FEBbrqok3UWBz SoISB3EOCB+6dxfOcfm9i+/FRbSfbzd3H6+LcJOl1Fzyoc0/bZfM5BHcCaMkharZ RYp0wfV2raUAiK8DI1xTTFyf380V1KG1MOa6z/jnfneW5sas05OumblbeiGQBV8q 8ZOcJ8qMCYGfw5DMeNjNXZC2MlQuCRkI0B3xr7kVYliZf7Tz2A5xAXC2W7cYAQDE 4VcD6CpYkGZx9/xG1pL4RGc+qUTgCRai9MXV3pVKIc+LAMYHVh4mhYp+iNHeR/8M o8Wn+TueQOjOLp4PeVSTHIHoCRdAZAEySXAPpvmonEvHENgSecUhAEhwtNqqXSTA xRelSG88bfT+LIRIiB4yCmpA/Wctz0D8naSk7VV7PgExKv8yxBswGo5gbXht2byp 4j9jhBn2RQQWDxIU4qoBaD717N0lhnZYtquDAFGSjjMddMK/Ut8TTBXj5/7qZhNQ Gx6szkowKpg+elFbGHexAysT+HJ+rcbnvGtscvvsjmdZWY8FjtYmbChbbMATBDNM FazT8CXqkivy2i/YbmLyLgUzTDo8SlayumAnZZemHZcPz/kfCk+ZT3sAZjtuw+Jy RmqZ5APR9vpEW3Bkb3Hdmc2LqZo= =nJ1T -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
On 22.12.2017 13:48, Eric Robinson wrote: We have multiple JVMs deployed on two identical Linux servers. Each server has 60 JVMs. Until today, both servers were running Tomcat6 with JDK 1.6. Today we upgraded one of the servers to Tomcat 8 with JDK 1.8. Now the JVMs on the Tomcat 8 server are each using between 20-80% more memory than the ones on Tomcat6 with JDK 1.6. Is that normal? Why would that be? Is it some kind of settings? Is it fixable? It might be as simple as different thresholds for the garbage collector to kick in. I'd start with an evaluation of how much memory is used right after a GC run - and in case this isn't satisfactory, which objects use the memory. Typically it's appropriate to just look at the top of the list. Note that the GC algorithms (or just GC defaults) between the different JVM versions (sometimes even between minor upgrades) might differ significantly. One big question is: Did you explicitly configure memory consumption, GC algorithm and thresholds? If so, what's the difference between the two options: I'd expect that you need to change the settings significantly in order to achieve the same behavior. There's a lot of work that has been done in this world. That being said, I'd also not rule out that tomcat's or other component's implementation changed - e.g. caches, or just memory use through upgraded libraries. But I'd recommend to look in both directions, with JDK and GC tuning being the elephant in the room, giving you the biggest bang for your buck. Olaf - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Is it Normal for Tomcat 8 to Use 20-80% More Memory Than Tomcat 6?
We have multiple JVMs deployed on two identical Linux servers. Each server has 60 JVMs. Until today, both servers were running Tomcat6 with JDK 1.6. Today we upgraded one of the servers to Tomcat 8 with JDK 1.8. Now the JVMs on the Tomcat 8 server are each using between 20-80% more memory than the ones on Tomcat6 with JDK 1.6. Is that normal? Why would that be? Is it some kind of settings? Is it fixable? --Eric - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org