Re: CentOS Tomcat install seems to ignore setenv.sh
On Wed, May 13, 2020 at 5:06 PM Patrick Baldwin wrote: > On Wed, May 13, 2020 at 1:31 PM Coty Sutherland > wrote: > > > Hi, > > > > Please see responses in line below. I'm top posting a bit because the > > thread got off in the weeds about permissions it seems, which are > important > > but not exactly relevant to your problem IMO. > > > > > Indeed, thank you. > > > > On Tue, May 12, 2020 at 11:28 AM Patrick Baldwin < > > pbald...@myersinfosys.com> > > wrote: > > > > > I've gotten passed an odd (to me, anyway) issue with one of our clients > > > CentOS systems. > > > > > > When our webapp starts running, tomcat dies shortly thereafter with an > > > OutOfMemoryError. This apparently just started a few days ago. > > > > > > > The issue isn't really odd. The JVM is telling you that something is > > preventing the garbage collector from being effective and therefore > > exhausting your heap space. See > > > > > https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html#parallel_collector_excessive_gc > > for more information about the particular OOME you noted that you're > > experiencing. > > > > > Reading that, I'm not quite sure if this error would happen if tomcat is > honoring the memory restrictions that have been (hopefully?) set in config. > One of our devs thinks the issue is with tomcat not honoring memory > restrictions, so I'm trying to see if there's any way I can make sure it > is. > > I'm also trying to figure out if this could be an issue with the Java code, > and not tomcat config per se. > > > > > > > System info: > > > > > > Tomcat Version: Apache Tomcat/7.0.76 > > > > > > JVM version: 1.8.0_191-b12 > > > > > > OS: CentOS Linux release 7.6.1810 (Core) > > > > > > > > > This seemed to indicate that catalina.sh isn’t the place for > environment > > > variables on Tomcat 7 for Linux: > > > > > > https://forums.centos.org/viewtopic.php?t=54207 > > > > > > > > > Since there isn’t a setenv.sh in /usr/local/tomcat/bin, we create one: > > > > > > > > > https://stackoverflow.com/questions/9480210/tomcat-7-setenv-sh-is-not-found > > > > > > 195$ ls -l /usr/local/tomcat/bin/setenv.sh > > > > > > -rwxrwxrwx. 1 root tomcat 110 May 11 12:56 > > /usr/local/tomcat/bin/setenv.sh > > > > > > 45$ cat /usr/local/tomcat/bin/setenv.sh > > > > > > > Assuming you've installed tomcat using yum, the startup doesn't use > > startup.sh at all so the setenv.sh script is ignored. Instead you want to > > put your settings into /etc/tomcat/tomcat.conf which is sourced by the > > systemd service unit. If you want to learn more about how that works, > check > > out the unit file to see which scripts it calls > (/usr/libexec/tomcat/server > > -> /usr/libexec/tomcat/preamble -> /usr/libexec/tomcat/functions). > > > > > > > To /etc/tomcat/tomcat.conf I added: > > # You can pass some parameters to java here if you wish to > #JAVA_OPTS="-Xminf0.1 -Xmaxf0.3" > > JAVA_OPTS="-Xmx2048m -XX:MaxPermSize=2048m" > > And now see: > > sudo journalctl -u tomcat -f : > > May 13 15:50:01 protrack server[24306]: OpenJDK 64-Bit Server VM warning: > ignoring option MaxPermSize=2048m; support was removed in 8.0 > > ... > > May 13 15:50:01 protrack server[24306]: INFO: Command line argument: > -Xmx2048m > > May 13 15:50:01 protrack server[24306]: May 13, 2020 3:50:01 PM > org.apache.catalina.startup.VersionLoggerListener log > > May 13 15:50:01 protrack server[24306]: INFO: Command line argument: > -XX:MaxPermSize=2048m > > May 13 15:50:01 protrack server[24306]: May 13, 2020 3:50:01 PM > org.apache.catalina.startup.VersionLoggerListener log > > May 13 15:50:01 protrack server[24306]: INFO: Command line argument: > -Xms2048m > > May 13 15:50:01 protrack server[24306]: May 13, 2020 3:50:01 PM > org.apache.catalina.startup.VersionLoggerListener log > > May 13 15:50:01 protrack server[24306]: INFO: Command line argument: > -Xmx2048m > > ... > > May 13 15:51:23 protrack server[24306]: SEVERE: Unexpected death of > background thread ContainerBackgroundProcessor[StandardEngine[Catalina]] > > May 13 15:51:23 protrack server[24306]: java.lang.OutOfMemoryError: GC > overhead limit exceeded > > May 13 15:51:23 protrack server[24306]: Exception in thread > "ContainerBackgroundProcessor[StandardEngine[Catalina]]" > java.lang.OutOfMemoryError: GC overhead limit exceeded > >From this you can see that the options are being added (Xmx is there twice, no problem though since the value is the same), and then you hit an OOME within 1.5 minutes. Have you tried increasing the heap size to see if it helps at all? > So, it is now definitely picking up that memory restriction, but it seems > to be ignoring because it’s deprecated. > > I’ve also found this: > > > https://stackoverflow.com/questions/22634644/java-hotspottm-64-bit-server-vm-warning-ignoring-option-maxpermsize > > Specifically, “I think this was downvoted because it implies that you > should switch previous uses of MaxPermGen with MaxMetaSpaceSize which is > misleading, since
Re: CentOS Tomcat install seems to ignore setenv.sh
On Wed, May 13, 2020 at 1:31 PM Coty Sutherland wrote: > Hi, > > Please see responses in line below. I'm top posting a bit because the > thread got off in the weeds about permissions it seems, which are important > but not exactly relevant to your problem IMO. > > Indeed, thank you. > On Tue, May 12, 2020 at 11:28 AM Patrick Baldwin < > pbald...@myersinfosys.com> > wrote: > > > I've gotten passed an odd (to me, anyway) issue with one of our clients > > CentOS systems. > > > > When our webapp starts running, tomcat dies shortly thereafter with an > > OutOfMemoryError. This apparently just started a few days ago. > > > > The issue isn't really odd. The JVM is telling you that something is > preventing the garbage collector from being effective and therefore > exhausting your heap space. See > > https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html#parallel_collector_excessive_gc > for more information about the particular OOME you noted that you're > experiencing. > > Reading that, I'm not quite sure if this error would happen if tomcat is honoring the memory restrictions that have been (hopefully?) set in config. One of our devs thinks the issue is with tomcat not honoring memory restrictions, so I'm trying to see if there's any way I can make sure it is. I'm also trying to figure out if this could be an issue with the Java code, and not tomcat config per se. > > > System info: > > > > Tomcat Version: Apache Tomcat/7.0.76 > > > > JVM version: 1.8.0_191-b12 > > > > OS: CentOS Linux release 7.6.1810 (Core) > > > > > > This seemed to indicate that catalina.sh isn’t the place for environment > > variables on Tomcat 7 for Linux: > > > > https://forums.centos.org/viewtopic.php?t=54207 > > > > > > Since there isn’t a setenv.sh in /usr/local/tomcat/bin, we create one: > > > > > https://stackoverflow.com/questions/9480210/tomcat-7-setenv-sh-is-not-found > > > > 195$ ls -l /usr/local/tomcat/bin/setenv.sh > > > > -rwxrwxrwx. 1 root tomcat 110 May 11 12:56 > /usr/local/tomcat/bin/setenv.sh > > > > 45$ cat /usr/local/tomcat/bin/setenv.sh > > > > Assuming you've installed tomcat using yum, the startup doesn't use > startup.sh at all so the setenv.sh script is ignored. Instead you want to > put your settings into /etc/tomcat/tomcat.conf which is sourced by the > systemd service unit. If you want to learn more about how that works, check > out the unit file to see which scripts it calls (/usr/libexec/tomcat/server > -> /usr/libexec/tomcat/preamble -> /usr/libexec/tomcat/functions). > > > To /etc/tomcat/tomcat.conf I added: # You can pass some parameters to java here if you wish to #JAVA_OPTS="-Xminf0.1 -Xmaxf0.3" JAVA_OPTS="-Xmx2048m -XX:MaxPermSize=2048m" And now see: sudo journalctl -u tomcat -f : May 13 15:50:01 protrack server[24306]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=2048m; support was removed in 8.0 ... May 13 15:50:01 protrack server[24306]: INFO: Command line argument: -Xmx2048m May 13 15:50:01 protrack server[24306]: May 13, 2020 3:50:01 PM org.apache.catalina.startup.VersionLoggerListener log May 13 15:50:01 protrack server[24306]: INFO: Command line argument: -XX:MaxPermSize=2048m May 13 15:50:01 protrack server[24306]: May 13, 2020 3:50:01 PM org.apache.catalina.startup.VersionLoggerListener log May 13 15:50:01 protrack server[24306]: INFO: Command line argument: -Xms2048m May 13 15:50:01 protrack server[24306]: May 13, 2020 3:50:01 PM org.apache.catalina.startup.VersionLoggerListener log May 13 15:50:01 protrack server[24306]: INFO: Command line argument: -Xmx2048m ... May 13 15:51:23 protrack server[24306]: SEVERE: Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]] May 13 15:51:23 protrack server[24306]: java.lang.OutOfMemoryError: GC overhead limit exceeded May 13 15:51:23 protrack server[24306]: Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: GC overhead limit exceeded So, it is now definitely picking up that memory restriction, but it seems to be ignoring because it’s deprecated. I’ve also found this: https://stackoverflow.com/questions/22634644/java-hotspottm-64-bit-server-vm-warning-ignoring-option-maxpermsize Specifically, “I think this was downvoted because it implies that you should switch previous uses of MaxPermGen with MaxMetaSpaceSize which is misleading, since their roles have practically reversed. Before Java 8 class metadata space resided in PermGen which was limited by 32/64MB, and MaxPerGen was used to increase it. Starting from Java 8 however, PermGen is no more and class metadata space is unlimited, so MaxMetaspace size is actually used to decrease it. “ So that seems to suggest Java is not running out of heap memory and making tomcat die, but that it really is just spending a lot of time on garbage collection? > > export CATALINA_OPTS="-server -Xms2048m -Xmx2048m" > > > > export
Re: CentOS Tomcat install seems to ignore setenv.sh
Hi, Please see responses in line below. I'm top posting a bit because the thread got off in the weeds about permissions it seems, which are important but not exactly relevant to your problem IMO. On Tue, May 12, 2020 at 11:28 AM Patrick Baldwin wrote: > I've gotten passed an odd (to me, anyway) issue with one of our clients > CentOS systems. > > When our webapp starts running, tomcat dies shortly thereafter with an > OutOfMemoryError. This apparently just started a few days ago. > The issue isn't really odd. The JVM is telling you that something is preventing the garbage collector from being effective and therefore exhausting your heap space. See https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html#parallel_collector_excessive_gc for more information about the particular OOME you noted that you're experiencing. > System info: > > Tomcat Version: Apache Tomcat/7.0.76 > > JVM version: 1.8.0_191-b12 > > OS: CentOS Linux release 7.6.1810 (Core) > > > This seemed to indicate that catalina.sh isn’t the place for environment > variables on Tomcat 7 for Linux: > > https://forums.centos.org/viewtopic.php?t=54207 > > > Since there isn’t a setenv.sh in /usr/local/tomcat/bin, we create one: > > https://stackoverflow.com/questions/9480210/tomcat-7-setenv-sh-is-not-found > > 195$ ls -l /usr/local/tomcat/bin/setenv.sh > > -rwxrwxrwx. 1 root tomcat 110 May 11 12:56 /usr/local/tomcat/bin/setenv.sh > > 45$ cat /usr/local/tomcat/bin/setenv.sh > Assuming you've installed tomcat using yum, the startup doesn't use startup.sh at all so the setenv.sh script is ignored. Instead you want to put your settings into /etc/tomcat/tomcat.conf which is sourced by the systemd service unit. If you want to learn more about how that works, check out the unit file to see which scripts it calls (/usr/libexec/tomcat/server -> /usr/libexec/tomcat/preamble -> /usr/libexec/tomcat/functions). > export CATALINA_OPTS="-server -Xms2048m -Xmx2048m" > > export JAVA_OPTS="-XX:PermSize=256m -XX:MaxPermSize=2048m" > > 46$ > > > System memory before starting tomcat: > > 188$ free -h > > totalusedfree shared buff/cache > available > > Mem:11G2.3G2.2G2.0G7.1G > 6.7G > > Swap: 8.0G1.0G7.0G > > > Started tomcat, with sudo service tomcat start > > Tomcat journal error: > > > May 11 17:48:59 protrack server[7298]: SEVERE: Unexpected death of > background thread ContainerBackgroundProcessor[StandardEngine[Catalina]] > > May 11 17:48:59 protrack server[7298]: java.lang.OutOfMemoryError: GC > overhead limit exceeded > > May 11 17:48:59 protrack server[7298]: Exception in thread > "ContainerBackgroundProcessor[StandardEngine[Catalina]]" > java.lang.OutOfMemoryError: GC overhead limit exceeded > > May 11 17:49:38 protrack server[7298]: Exception: > java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in > thread "http-bio-8080-AsyncTimeout" > > May 11 17:49:39 protrack server[7298]: Exception: > java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in > thread "ajp-bio-8009-AsyncTimeout" > > May 11 17:49:42 protrack server[7298]: Exception in thread > > "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread" > > > Application log error: > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-05-11 17:49:50 > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2] > ERROR o.s.t.i.TransactionInterceptor - Application exception overridden by > rollback exception > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > System memory while tomcat is up, after the OutOfMemoryError pops: > > ksmq_tv 191$ free -h > > totalusedfree shared buff/cache > available > > Mem:11G3.5G1.0G2.0G7.1G > 5.5G > > Swap: 8.0G1.0G7.0G > > > Stopped with sudo service tomcat stop > > > > System memory after tomcat stopped: > > ksmq_tv 194$ free -h > > totalusedfree shared buff/cache > available > > Mem:11G795M3.7G2.0G7.1G > 8.2G > > Swap: 8.0G1.0G7.0G > > > > It sure doesn't look like something is actually running the system out of > memory at a system level; usage is definitely impacted by starting our app, > but that's expected. > The system isn't running out of memory, Tomcat's JVM is. This could be due to numerous things, so you'll have to do some digging to find out why that is. I'd start by enabling/collecting/reviewing GC logging and a heap dump from the time of the OOME, which you may have to take manually (I don't recall if the HeapDumpOnOutOfMemory argument triggers with a GC overhead error). As a simple solution try and increase the amount of heap that you give the instance to see if the problem goes away or if it occurs after a
Re: [OT] Re: CentOS Tomcat install seems to ignore setenv.sh
Chris, > Am 13.05.2020 um 16:42 schrieb Christopher Schultz > : > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Peter, > > On 5/13/20 02:48, logo wrote: >> Hi calder, >> >> >>> Am 13.05.2020 um 04:59 schrieb calder : >>> >>> On Tue, May 12, 2020, 21:48 kohmoto >>> wrote: >>> Hi, Calder, Thank you for your prompt reply. I think Tomcat binary files all have root priviledges. Should these priviledges should be changed to user priviledges? >>> >>> >>> Yes. >> >> I would suggest to leave the binaries and maybe even config files >> to root or any other admin. So a hacked tomcat process under tomcat >> user will not be able to exchange config or even binaries. > Yes! There really is no need for Tomcat to modify its own config files > or binaries. > >> That will only work if the config will not be changed via >> host-manager or programmatically. > > This shouldn't be too much of a problem. Not many people use the > host-manager. > >> In the past we even held the installed webapps under a different >> user. but that maybe difficult in automated deployments. > > This is less important IMO. The owner of the files can be anybody... > just not the Tomcat user. > > And, before anybody says "but.. but... Docker!" you should remember > that root in a Docker container often ends up having many more > privileges outside the container than you think it does/should. > Never ignore this! As simple as adding the following to your Dockerfile RUN set -x \ && groupadd tomcat \ && useradd -g tomcat -s /usr/bin/nologin -m -d /home/tomcat tomcat \ && chown -R tomcat:tomcat $CATALINA_HOME/logs $CATALINA_HOME/work $CATALINA_HOME/temp # add $CATALINA_HOME/webapps if you use the manager-app USER tomcat before running the ENTRYPOINT or CMD. BTW: that is something that is really missing in the „Official“ Tomcat Docker images. (I know they are not maintained by ASF) Peter > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl68B08ACgkQHPApP6U8 > pFhlGA/+O68ss6u7JTG4j7LDOqZq3K+E6DJ+45v8Tz69nL49WNZzcs3n1DfhP3Ar > EomgYcbGgAq0eot6LInWQmoMX1xp+Wh2XC0b/fae1/vkerN4ugqLiqWIA1uM1ar4 > LQ0im7X8RaredMc0BanditDuJK5iRoUqRg7md8Sw/aGOncvVzjRR1hgU9mzqbHps > C6wrSb+5rr5a3SE1P2k2uhpMY1FI1xP8icdQXBCA6eYiJ3tBYsPmWU3z3J8JAmmG > k3i4WpZdoTsfVlX2n8H6K1l3u8WQsHmoMg5pzqgV1ABUtSBBbrSSkdCTlwHse4ch > ePAZg5XRuVlb06UbRHW15hfVGJCgCrzeL/yIufScEOYV6YYnEAx5D+UULG0wSJ7r > cRnmPE5ccWSLflplcmhurqhXRWeqsfpzjraZGRdyqv4HqG1zlfefEIuMVCPLHtRO > DQACi0HfbhOvNoIsVtabXBC4gigT4Zv2k9EQcRjbpO/cN1VMzRNV80jVnnV4QiOj > PCJ81Z4G+LE0pCSqoYLIvZKZ1CbmD0BIeL6ZngrHJFACJlkbSK4zfRMBgKc/2OZa > 5ltCftRev8fW4hztptQRxg9tfEdhaa+6MVXTGPTHYhGycMsDgCEO/W7dU6LcJNJG > q+bcNedNE5y046yCm9SpGKW0/kmS5I6R984W1NhfHSNKTLtEwHc= > =V1n4 > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: CentOS Tomcat install seems to ignore setenv.sh
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Peter, On 5/13/20 02:48, logo wrote: > Hi calder, > > >> Am 13.05.2020 um 04:59 schrieb calder : >> >> On Tue, May 12, 2020, 21:48 kohmoto >> wrote: >> >>> Hi, Calder, >>> >>> Thank you for your prompt reply. I think Tomcat binary files >>> all have root priviledges. Should these priviledges should be >>> changed to user priviledges? >>> >> >> >> Yes. > > I would suggest to leave the binaries and maybe even config files > to root or any other admin. So a hacked tomcat process under tomcat > user will not be able to exchange config or even binaries. Yes! There really is no need for Tomcat to modify its own config files or binaries. > That will only work if the config will not be changed via > host-manager or programmatically. This shouldn't be too much of a problem. Not many people use the host-manager. > In the past we even held the installed webapps under a different > user. but that maybe difficult in automated deployments. This is less important IMO. The owner of the files can be anybody... just not the Tomcat user. And, before anybody says "but.. but... Docker!" you should remember that root in a Docker container often ends up having many more privileges outside the container than you think it does/should. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl68B08ACgkQHPApP6U8 pFhlGA/+O68ss6u7JTG4j7LDOqZq3K+E6DJ+45v8Tz69nL49WNZzcs3n1DfhP3Ar EomgYcbGgAq0eot6LInWQmoMX1xp+Wh2XC0b/fae1/vkerN4ugqLiqWIA1uM1ar4 LQ0im7X8RaredMc0BanditDuJK5iRoUqRg7md8Sw/aGOncvVzjRR1hgU9mzqbHps C6wrSb+5rr5a3SE1P2k2uhpMY1FI1xP8icdQXBCA6eYiJ3tBYsPmWU3z3J8JAmmG k3i4WpZdoTsfVlX2n8H6K1l3u8WQsHmoMg5pzqgV1ABUtSBBbrSSkdCTlwHse4ch ePAZg5XRuVlb06UbRHW15hfVGJCgCrzeL/yIufScEOYV6YYnEAx5D+UULG0wSJ7r cRnmPE5ccWSLflplcmhurqhXRWeqsfpzjraZGRdyqv4HqG1zlfefEIuMVCPLHtRO DQACi0HfbhOvNoIsVtabXBC4gigT4Zv2k9EQcRjbpO/cN1VMzRNV80jVnnV4QiOj PCJ81Z4G+LE0pCSqoYLIvZKZ1CbmD0BIeL6ZngrHJFACJlkbSK4zfRMBgKc/2OZa 5ltCftRev8fW4hztptQRxg9tfEdhaa+6MVXTGPTHYhGycMsDgCEO/W7dU6LcJNJG q+bcNedNE5y046yCm9SpGKW0/kmS5I6R984W1NhfHSNKTLtEwHc= =V1n4 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Re: CentOS Tomcat install seems to ignore setenv.sh
Hi, Logo, My current practice is as you suggest. Thank you for your advice. Yours truly, Kazuhiko Kohmoto On 2020/05/13 15:48, logo wrote: Hi calder, Am 13.05.2020 um 04:59 schrieb calder : On Tue, May 12, 2020, 21:48 kohmoto wrote: Hi, Calder, Thank you for your prompt reply. I think Tomcat binary files all have root priviledges. Should these priviledges should be changed to user priviledges? Yes. I would suggest to leave the binaries and maybe even config files to root or any other admin. So a hacked tomcat process under tomcat user will not be able to exchange config or even binaries. That will only work if the config will not be changed via host-manager or programmatically. In the past we even held the installed webapps under a different user. but that maybe difficult in automated deployments. My 2cts. Peter There is a "Tomcat Security" guide at the Tomcat website. Also, Mulesoft has a good guide https://www.mulesoft.com/tcat/tomcat-security Your truly, Kazuhiko Kohmoto On 2020/05/13 11:17, calder wrote: If TC, running as root, is ever compromised, the compromising user (attacker) can gain access to the whole of the system. The attacker could execute any arbitrary command available on the system. They could remove files, or install malicious software. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[OT] Re: CentOS Tomcat install seems to ignore setenv.sh
Hi calder, > Am 13.05.2020 um 04:59 schrieb calder : > > On Tue, May 12, 2020, 21:48 kohmoto wrote: > >> Hi, Calder, >> >> Thank you for your prompt reply. >> I think Tomcat binary files all have root priviledges. >> Should these priviledges should be changed to user priviledges? >> > > > Yes. I would suggest to leave the binaries and maybe even config files to root or any other admin. So a hacked tomcat process under tomcat user will not be able to exchange config or even binaries. That will only work if the config will not be changed via host-manager or programmatically. In the past we even held the installed webapps under a different user. but that maybe difficult in automated deployments. My 2cts. Peter > > There is a "Tomcat Security" guide at the Tomcat website. Also, Mulesoft > has a good guide > https://www.mulesoft.com/tcat/tomcat-security > > > Your truly, >> Kazuhiko Kohmoto >> >> On 2020/05/13 11:17, calder wrote: >>> If TC, running as root, is ever compromised, the compromising user >>> (attacker) can gain access to the whole of the system. The attacker >> could >>> execute any arbitrary command available on the system. They could remove >>> files, or install malicious software. >> >> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
Hi, Calder, Thank you again for your prompt reply. I will study these documents you kindly shows. Thank you. Yours truly, Kazuhiko Kohmoto On 2020/05/13 11:59, calder wrote: Yes. There is a "Tomcat Security" guide at the Tomcat website. Also, Mulesoft has a good guide https://www.mulesoft.com/tcat/tomcat-security - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020, 21:48 kohmoto wrote: > Hi, Calder, > > Thank you for your prompt reply. > I think Tomcat binary files all have root priviledges. > Should these priviledges should be changed to user priviledges? > Yes. There is a "Tomcat Security" guide at the Tomcat website. Also, Mulesoft has a good guide https://www.mulesoft.com/tcat/tomcat-security Your truly, > Kazuhiko Kohmoto > > On 2020/05/13 11:17, calder wrote: > > If TC, running as root, is ever compromised, the compromising user > > (attacker) can gain access to the whole of the system. The attacker > could > > execute any arbitrary command available on the system. They could remove > > files, or install malicious software. > >
Re: CentOS Tomcat install seems to ignore setenv.sh
Hi, Calder, Thank you for your prompt reply. I think Tomcat binary files all have root priviledges. Should these priviledges should be changed to user priviledges? Your truly, Kazuhiko Kohmoto On 2020/05/13 11:17, calder wrote: If TC, running as root, is ever compromised, the compromising user (attacker) can gain access to the whole of the system. The attacker could execute any arbitrary command available on the system. They could remove files, or install malicious software. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020, 19:58 kohmoto wrote: > > On 2020/05/13 0:47, John Larsen wrote: > > I wouldnt recommend running tomcat as root > > Actually I run Tomcat as root. Your recommendation seems > against my practice. It would be appreciated if you would > advice me about points not running as root This is not a security practice only for Tomcat, but ANY application running on a Linux system (or any operating system for that matter). If TC, running as root, is ever compromised, the compromising user (attacker) can gain access to the whole of the system. The attacker could execute any arbitrary command available on the system. They could remove files, or install malicious software. To limit a compromise, one should run the user of the application with the least privileges (never as the root user) Read more here https://en.m.wikipedia.org/wiki/Principle_of_least_privilege Limiting privileges can also be beneficial for lessening any inadvertent damage, for example, to the local filesystem. Let's say the app creates, reads, writes, and deletes files. And let's say the app is about to delete all files in a subdirectory path, but the URI gets munged and the path to be purged is a system path. Because the app has root privileges, all those system files will be removed.If the TC app is running as a limited user (something traditional like "tomcat"), those system files will not be deleted.
Re: CentOS Tomcat install seems to ignore setenv.sh
Hi, John, Actually I run Tomcat as root. Your recommendation seems against my practice. It would be appreciated if you would advice me about points not running as root. Thank you. Yours truly, Kazuhiko Kohmto On 2020/05/13 0:47, John Larsen wrote: I wouldnt recommend running tomcat as root - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020 at 4:30 PM Patrick Baldwin wrote: > On Tue, May 12, 2020 at 5:13 PM Christopher Schultz < > ch...@christopherschultz.net> wrote: [snip] > > There is no catalina.sh that I can find. When I googled that, I found: > https://forums.centos.org/viewtopic.php?t=54207 > > You should be able to use systemd for this. > > If I can get it going that way, I'll definitely try it. No need to try ... it is running via Systemd : ) > 102$ sudo service tomcat start That's why catalina.sh isn't used. This might be helpful as a sort of "tutorial" https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-8-on-centos-7 Pay particular attention to the section, "Install Systemd Unit File" .. where they show a sample "/etc/systemd/system/tomcat.service" - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020 at 5:13 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Patrick, > > On 5/12/20 17:08, Patrick Baldwin wrote: > > 102$ sudo service tomcat start Redirecting to /bin/systemctl start > > tomcat.service > > Uhh, doesn't that just call systemd? > > Are you sure it's restarting at all? > > Yes, I can watch it go up and down in the logs. > If you launch Tomcat with catalina.sh or anything which has been > modelled after it, it will run setenv.sh for you. But systemd has > other plans. > > There is no catalina.sh that I can find. When I googled that, I found: https://forums.centos.org/viewtopic.php?t=54207 > You should be able to use systemd for this. You just need to convince > it to pass extra parameters to the JVM when it starts. Maybe Coty > (RedHat) or Emmanuel (Debian) can give some advice on this. I'm no > systemd expert. > > If I can get it going that way, I'll definitely try it. It's just odd this was running for months with this config using setenv.sh and now this has apparently spontaneously started. > - -chris > > > On Tue, May 12, 2020 at 5:07 PM calder > > wrote: > > > >> On Tue, May 12, 2020, 15:49 Patrick Baldwin > >> wrote: > >> > >>> I turned off systemd for tomcat: > >>> > >>> 84$ sudo systemctl disable tomcat > >> > >> > >> > >> [ snip ] > >> > >>> Restarted tomcat, and it still runs for about 2 minutes then > >>> throws > >> > >> that java.lang.OutOfMemoryError: Java heap space error and dies. > >> > >> > >> > >> And how are you starting TC, as now not using Systemd? > >> > > > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl67EX4ACgkQHPApP6U8 > pFhIyA/+LzV9Ta95tx5RKkBZVLS5e4iVjS06CLmp3qkyvzfNOU+ryXaWILSibpRA > 8Y7JV+M5AJjY6J/2nlXAddTivNVbtXBYAttuXJyWirM1RkEEGF53hVrxcj+FK/zD > hCi1fxveBxCz63txBqy6La16Xh3qcK9b0l5PX7iAQVsPuv0hPxkCvg0maxFtlyiA > DNrtFFdlNlPCvhXR8nAia7RLp02wbBUG8FdlN/MUqi6YEzwq6H0uYbiz9EE/3aMa > IL1Kr2M/fm6eZl2OmECvc+Lh+vFgYvK4H4tWH+z6irUwYOXNFmJAm6ulU8XTn5ox > NYU4aPlw7Y1Ez0FYBxohH6D4cA6DA+8rH8d2Aniw4Xj/LQNiZOxJwfYAYJvFP78H > okwbBCr9HAvS96uQo7Y1znq9t+v0rG44lVu7l0gYAmC8yHQnHDUyf9k6OQ0ILZ5a > LQorjrJZjbQ+i1m6amZIpzua3bOGX69rHtsxwZCj0KL1FQdhW4LPMJrJ5/SdSo7+ > ryO6dMzpJtKBMJeYjPIOCFXNickDlPc9cwbyn5Bow4gAwkDRjn1WduyNLHSdjK+U > DPTZULtLyaOYWDWT0T28LWha+911b6uoKgNvANPm4r6A51rt4rDz0gbDeG/U5ygv > vLSaO0PHjcfJbN8dri9YMcUTl0fezXeaMrR9ae44y324NzaJKIk= > =ccOJ > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020, 16:13 Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Patrick, > > On 5/12/20 17:08, Patrick Baldwin wrote: > > 102$ sudo service tomcat start Redirecting to /bin/systemctl start > > tomcat.service > > Uhh, doesn't that just call systemd? > Agreed. In my previous post, I mentioned the config file to use ... (Systemd is still in use.) [ previous quote ] Does their Tomcat use Systemd? If yes, then look for the Systemd unit file - default should be /etc/systemd/system/tomcat.service that's where they will add an entry like (or modify the existing) [ ... ] Environment='CATALINA_OPTS=-server -Xms2048m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=2048m' [ ... ] The memory settings need to go in /etc/systemd/system/tomcat.service (among othe JVM parms)
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020, 16:08 Patrick Baldwin wrote: > On Tue, May 12, 2020 at 5:07 PM calder wrote: > > > On Tue, May 12, 2020, 15:49 Patrick Baldwin > > wrote: > > > > > I turned off systemd for tomcat: > > > > > > 84$ sudo systemctl disable tomcat > > > > [ snip ] > > > > > Restarted tomcat, and it still runs for about 2 minutes then throws > > > > that java.lang.OutOfMemoryError: Java heap space error and dies. > > > > > > > > And how are you starting TC, as now not using Systemd? > 102$ sudo service tomcat start > Redirecting to /bin/systemctl start tomcat.service But setenv.sh is sourced by catalina.sh. so if it isn't executed, setenv.sh won't.
Re: CentOS Tomcat install seems to ignore setenv.sh
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Patrick, On 5/12/20 17:08, Patrick Baldwin wrote: > 102$ sudo service tomcat start Redirecting to /bin/systemctl start > tomcat.service Uhh, doesn't that just call systemd? Are you sure it's restarting at all? If you launch Tomcat with catalina.sh or anything which has been modelled after it, it will run setenv.sh for you. But systemd has other plans. You should be able to use systemd for this. You just need to convince it to pass extra parameters to the JVM when it starts. Maybe Coty (RedHat) or Emmanuel (Debian) can give some advice on this. I'm no systemd expert. - -chris > On Tue, May 12, 2020 at 5:07 PM calder > wrote: > >> On Tue, May 12, 2020, 15:49 Patrick Baldwin >> wrote: >> >>> I turned off systemd for tomcat: >>> >>> 84$ sudo systemctl disable tomcat >> >> >> >> [ snip ] >> >>> Restarted tomcat, and it still runs for about 2 minutes then >>> throws >> >> that java.lang.OutOfMemoryError: Java heap space error and dies. >> >> >> >> And how are you starting TC, as now not using Systemd? >> > -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl67EX4ACgkQHPApP6U8 pFhIyA/+LzV9Ta95tx5RKkBZVLS5e4iVjS06CLmp3qkyvzfNOU+ryXaWILSibpRA 8Y7JV+M5AJjY6J/2nlXAddTivNVbtXBYAttuXJyWirM1RkEEGF53hVrxcj+FK/zD hCi1fxveBxCz63txBqy6La16Xh3qcK9b0l5PX7iAQVsPuv0hPxkCvg0maxFtlyiA DNrtFFdlNlPCvhXR8nAia7RLp02wbBUG8FdlN/MUqi6YEzwq6H0uYbiz9EE/3aMa IL1Kr2M/fm6eZl2OmECvc+Lh+vFgYvK4H4tWH+z6irUwYOXNFmJAm6ulU8XTn5ox NYU4aPlw7Y1Ez0FYBxohH6D4cA6DA+8rH8d2Aniw4Xj/LQNiZOxJwfYAYJvFP78H okwbBCr9HAvS96uQo7Y1znq9t+v0rG44lVu7l0gYAmC8yHQnHDUyf9k6OQ0ILZ5a LQorjrJZjbQ+i1m6amZIpzua3bOGX69rHtsxwZCj0KL1FQdhW4LPMJrJ5/SdSo7+ ryO6dMzpJtKBMJeYjPIOCFXNickDlPc9cwbyn5Bow4gAwkDRjn1WduyNLHSdjK+U DPTZULtLyaOYWDWT0T28LWha+911b6uoKgNvANPm4r6A51rt4rDz0gbDeG/U5ygv vLSaO0PHjcfJbN8dri9YMcUTl0fezXeaMrR9ae44y324NzaJKIk= =ccOJ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
102$ sudo service tomcat start Redirecting to /bin/systemctl start tomcat.service On Tue, May 12, 2020 at 5:07 PM calder wrote: > On Tue, May 12, 2020, 15:49 Patrick Baldwin > wrote: > > > I turned off systemd for tomcat: > > > > 84$ sudo systemctl disable tomcat > > > > [ snip ] > > > Restarted tomcat, and it still runs for about 2 minutes then throws > > that java.lang.OutOfMemoryError: Java heap space error and dies. > > > > And how are you starting TC, as now not using Systemd? >
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020, 15:49 Patrick Baldwin wrote: > I turned off systemd for tomcat: > > 84$ sudo systemctl disable tomcat [ snip ] > Restarted tomcat, and it still runs for about 2 minutes then throws that java.lang.OutOfMemoryError: Java heap space error and dies. And how are you starting TC, as now not using Systemd?
Re: CentOS Tomcat install seems to ignore setenv.sh
I turned off systemd for tomcat: 84$ sudo systemctl disable tomcat [sudo] password: Removed symlink /etc/systemd/system/multi-user.target.wants/tomcat.service. Verified the setenv.sh file: 93$ ls -l /usr/share/tomcat/bin/setenv.sh -rw-rw-r--. 1 root tomcat 110 May 11 12:56 /usr/share/tomcat/bin/setenv.sh 94$ cat /usr/share/tomcat/bin/setenv.sh export CATALINA_OPTS="-server -Xms2048m -Xmx2048m" export JAVA_OPTS="-XX:PermSize=256m -XX:MaxPermSize=2048m" Restarted tomcat, and it still runs for about 2 minutes then throws that java.lang.OutOfMemoryError: Java heap space error and dies. On Tue, May 12, 2020 at 4:36 PM calder wrote: > On Tue, May 12, 2020, 13:48 calder wrote: > > > > > [ snip ] > > > > Does their Tomcat use Systemd? > > If yes, then look for the Systemd unit file - default should be > > /etc/systemd/system/tomcat.service > > that's where they will add an entry like (or modify the existing) > > [ ... ] > > Environment='CATALINA_OPTS=-server -Xms2048m -Xmx2048m > > -XX:PermSize=256m -XX:MaxPermSize=2048m' > > [ ... ] > > > My post may have been a bit misleading with my comment about setenv.sh and > then the reference to Systemd. If using Systemd, setenv.sh will be > ignored, hence the possible use of default JVM memory config, and thus > OOME. >
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020, 13:48 calder wrote: > > [ snip ] > Does their Tomcat use Systemd? > If yes, then look for the Systemd unit file - default should be > /etc/systemd/system/tomcat.service > that's where they will add an entry like (or modify the existing) > [ ... ] > Environment='CATALINA_OPTS=-server -Xms2048m -Xmx2048m > -XX:PermSize=256m -XX:MaxPermSize=2048m' > [ ... ] My post may have been a bit misleading with my comment about setenv.sh and then the reference to Systemd. If using Systemd, setenv.sh will be ignored, hence the possible use of default JVM memory config, and thus OOME.
Re: CentOS Tomcat install seems to ignore setenv.sh
The permission change is a temporary one while we try and figure out why this isn't working. On Tue, May 12, 2020 at 4:07 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > John, Patrick, > > On 5/12/20 11:47, John Larsen wrote: > > Should be chmod 644 and also I wouldnt recommend running tomcat as > > root. > > It's not clear that Tomcaat is running as root, but it IS clear that > setenv.sh is writable by ANYBODY and likely run by either root or a > semi-privilged Tomcat used. That's GOT to be fixed. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl67AhMACgkQHPApP6U8 > pFi75RAAlaXyEmwpwB+ppq1LhjZ+d+fgywB8TlgxeiqTqOi9Ak9fSOrn++DaEgiG > PSIcqCMktxvwhtQAjRKXX+QIWH8G+ZVMQ7gVBIzEjwpAtHXNUkDPt+MLkqIyMQD9 > xDhec5VZgiK9cXeUVot8g1HD3AxbuyTrJZq3LN+7WSpyTNU7m+I1EwoSsAyIHNDf > VUdkPiiIE58yZNmnac0H9wOIgJbuJhiFzRS98oVGLNMCCVf0UY+kjXJBV42PC7OT > GFBbJ7WOYPqdX3rj3L1F/WzQa8OW8se3cvqzWCTHpX67thzRB5GjxyHQrgXJ9/Hq > ZjOdC4JIg4VEqEdKpW/1i7TJEohD6W/RgwGO3B1v3cXzk3+rQVWyxGYdN6J0y6ca > dAQ7EoZpna8XGIBnH3CVTGt4lUUtSjWmiJjTn8x1wu2TZtpcMWKHVTF/fJ96z/xp > ze1k68/rFhU6AxpSHZz0yZpCnjeOPKYpi8blwUKV3s8LoIZ3eHCrgHkumNORuumk > 4NKp1JHT4vwmB4HxgjIfygTa/hCjkl5Hnnd4GQmY2lNfX8vkYC67p9fcS5iQaZmr > rhtre+ifF6bHG+1mCDORNXA0ZyELCLypxj97WHHt0BRv6D7Y8Qx/BzTrv7iNkM0X > /nfpv9MNO5ZrO6IYnNa2YCIeDgomEh9iroBwsLcuXWtPM0Hc9Aw= > =pz5d > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: CentOS Tomcat install seems to ignore setenv.sh
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John, Patrick, On 5/12/20 11:47, John Larsen wrote: > Should be chmod 644 and also I wouldnt recommend running tomcat as > root. It's not clear that Tomcaat is running as root, but it IS clear that setenv.sh is writable by ANYBODY and likely run by either root or a semi-privilged Tomcat used. That's GOT to be fixed. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl67AhMACgkQHPApP6U8 pFi75RAAlaXyEmwpwB+ppq1LhjZ+d+fgywB8TlgxeiqTqOi9Ak9fSOrn++DaEgiG PSIcqCMktxvwhtQAjRKXX+QIWH8G+ZVMQ7gVBIzEjwpAtHXNUkDPt+MLkqIyMQD9 xDhec5VZgiK9cXeUVot8g1HD3AxbuyTrJZq3LN+7WSpyTNU7m+I1EwoSsAyIHNDf VUdkPiiIE58yZNmnac0H9wOIgJbuJhiFzRS98oVGLNMCCVf0UY+kjXJBV42PC7OT GFBbJ7WOYPqdX3rj3L1F/WzQa8OW8se3cvqzWCTHpX67thzRB5GjxyHQrgXJ9/Hq ZjOdC4JIg4VEqEdKpW/1i7TJEohD6W/RgwGO3B1v3cXzk3+rQVWyxGYdN6J0y6ca dAQ7EoZpna8XGIBnH3CVTGt4lUUtSjWmiJjTn8x1wu2TZtpcMWKHVTF/fJ96z/xp ze1k68/rFhU6AxpSHZz0yZpCnjeOPKYpi8blwUKV3s8LoIZ3eHCrgHkumNORuumk 4NKp1JHT4vwmB4HxgjIfygTa/hCjkl5Hnnd4GQmY2lNfX8vkYC67p9fcS5iQaZmr rhtre+ifF6bHG+1mCDORNXA0ZyELCLypxj97WHHt0BRv6D7Y8Qx/BzTrv7iNkM0X /nfpv9MNO5ZrO6IYnNa2YCIeDgomEh9iroBwsLcuXWtPM0Hc9Aw= =pz5d -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
On Tue, May 12, 2020 at 10:28 AM Patrick Baldwin wrote: > > I've gotten passed an odd (to me, anyway) issue with one of our clients > CentOS systems. > > When our webapp starts running, tomcat dies shortly thereafter with an > OutOfMemoryError. This apparently just started a few days ago. > > System info: > Tomcat Version: Apache Tomcat/7.0.76 > JVM version: 1.8.0_191-b12 > OS: CentOS Linux release 7.6.1810 (Core) [ snip ] > Since there isn’t a setenv.sh in /usr/local/tomcat/bin, we create one: > 195$ ls -l /usr/local/tomcat/bin/setenv.sh > -rwxrwxrwx. 1 root tomcat 110 May 11 12:56 /usr/local/tomcat/bin/setenv.sh ---^ have the client correct this (owner/runas root) > 45$ cat /usr/local/tomcat/bin/setenv.sh > export CATALINA_OPTS="-server -Xms2048m -Xmx2048m" > export JAVA_OPTS="-XX:PermSize=256m -XX:MaxPermSize=2048m" Does their Tomcat use Systemd? If yes, then look for the Systemd unit file - default should be /etc/systemd/system/tomcat.service that's where they will add an entry like (or modify the existing) [ ... ] Environment='CATALINA_OPTS=-server -Xms2048m -Xmx2048m -XX:PermSize=256m -XX:MaxPermSize=2048m' [ ... ] In a pinch, you can check the TC logs to see if the variables are being set. > System memory before starting tomcat: > 188$ free -h "free" doesn't provide a proper memory report for a JVM. If it were me, I'd run *jvisualvm* or *MissionControl* (jmc) or *jconsole*.and get a snapshot of the app. Another option is to use *jmap* or *jcmd* You could also add this option -XX:+HeapDumpOnOutOfMemoryError and optionally this if you want to specify the path of the dump -XX:HeapDumpPath= ... if you feel like you cant get a proper capture with the GUI tools listed above [ snip ] - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CentOS Tomcat install seems to ignore setenv.sh
Should be chmod 644 and also I wouldnt recommend running tomcat as root. John Larsen On Tue, May 12, 2020 at 9:28 AM Patrick Baldwin wrote: > I've gotten passed an odd (to me, anyway) issue with one of our clients > CentOS systems. > > When our webapp starts running, tomcat dies shortly thereafter with an > OutOfMemoryError. This apparently just started a few days ago. > > System info: > > Tomcat Version: Apache Tomcat/7.0.76 > > JVM version: 1.8.0_191-b12 > > OS: CentOS Linux release 7.6.1810 (Core) > > > This seemed to indicate that catalina.sh isn’t the place for environment > variables on Tomcat 7 for Linux: > > https://forums.centos.org/viewtopic.php?t=54207 > > > Since there isn’t a setenv.sh in /usr/local/tomcat/bin, we create one: > > https://stackoverflow.com/questions/9480210/tomcat-7-setenv-sh-is-not-found > > 195$ ls -l /usr/local/tomcat/bin/setenv.sh > > -rwxrwxrwx. 1 root tomcat 110 May 11 12:56 /usr/local/tomcat/bin/setenv.sh > > 45$ cat /usr/local/tomcat/bin/setenv.sh > > export CATALINA_OPTS="-server -Xms2048m -Xmx2048m" > > export JAVA_OPTS="-XX:PermSize=256m -XX:MaxPermSize=2048m" > > 46$ > > > System memory before starting tomcat: > > 188$ free -h > > totalusedfree shared buff/cache > available > > Mem:11G2.3G2.2G2.0G7.1G > 6.7G > > Swap: 8.0G1.0G7.0G > > > Started tomcat, with sudo service tomcat start > > Tomcat journal error: > > > May 11 17:48:59 protrack server[7298]: SEVERE: Unexpected death of > background thread ContainerBackgroundProcessor[StandardEngine[Catalina]] > > May 11 17:48:59 protrack server[7298]: java.lang.OutOfMemoryError: GC > overhead limit exceeded > > May 11 17:48:59 protrack server[7298]: Exception in thread > "ContainerBackgroundProcessor[StandardEngine[Catalina]]" > java.lang.OutOfMemoryError: GC overhead limit exceeded > > May 11 17:49:38 protrack server[7298]: Exception: > java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in > thread "http-bio-8080-AsyncTimeout" > > May 11 17:49:39 protrack server[7298]: Exception: > java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in > thread "ajp-bio-8009-AsyncTimeout" > > May 11 17:49:42 protrack server[7298]: Exception in thread > > "org.springframework.scheduling.quartz.SchedulerFactoryBean#0_QuartzSchedulerThread" > > > Application log error: > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > 2020-05-11 17:49:50 > [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2] > ERROR o.s.t.i.TransactionInterceptor - Application exception overridden by > rollback exception > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > System memory while tomcat is up, after the OutOfMemoryError pops: > > ksmq_tv 191$ free -h > > totalusedfree shared buff/cache > available > > Mem:11G3.5G1.0G2.0G7.1G > 5.5G > > Swap: 8.0G1.0G7.0G > > > Stopped with sudo service tomcat stop > > > > System memory after tomcat stopped: > > ksmq_tv 194$ free -h > > totalusedfree shared buff/cache > available > > Mem:11G795M3.7G2.0G7.1G > 8.2G > > Swap: 8.0G1.0G7.0G > > > > It sure doesn't look like something is actually running the system out of > memory at a system level; usage is definitely impacted by starting our app, > but that's expected. > > Assuming no one finds any obvious errors with how we implemented setenv.sh, > is there some way to verify what memory limitations tomcat is actually > running under? > > I was also wondering if anyone knew an open source webapp that would be > good to deploy to see if this problem is tomcat specific or an issue with > our webapp? I figure if I deploy something else that doesn't promptly > throw an OutOfMemoryError, then it might be more of a dev issue and less > of a tomcat config issue. Trying to at least figure out what direction I > need to be looking in, any help much appreciated. >
