Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
On Thu, Jul 2, 2020 at 7:21 AM Coty Sutherland wrote: > The RHEL 7 Tomcat package uses systemd and the journal to capture stdout > instead of catalina.out. Did you check the journal to see if the thread > dumps are logged there from your invocations of kill -3? You can use > `journalctl -u tomcat` to check it. > > On Wed, Jul 1, 2020 at 6:58 PM Sean Neeley > wrote: > > > On Wed, Jul 1, 2020 at 5:24 PM calder wrote: > > > > > On Wed, Jul 1, 2020, 15:32 Sean Neeley > > wrote: > > > > > > > I tried switching from Java 1.8 to Java 11 to see if that makes a > > > > difference. Now the VM Thread is using a lot less CPU: > > > > > > > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ > > > COMMAND > > > > 2320 tomcat20 0 4659072 47872 19904 R 99.9 0.6 22:15.16 > java > > > > 2326 tomcat20 0 4659072 47872 19904 R 4.6 0.6 0:56.43 VM > > > > Thread > > > > > > > > I tried running jstack on the processes, but I get this: > > > > > > > > 2320: Unable to open socket file: target process not responding or > > > HotSpot > > > > VM not loaded > > > > > > > > > > Did you attempt to run the command as the "Tomcat user"? > > > > > > BTW, Oracle recommends the use of "jcmd" over "jstack". Personally, > I'd > > > give Mission Control/Flight Recorder a go. > > > > > > > I'm definitely running it as the tomcat user. I just tried jcmd with no > > arguments and the command completely hangs. The only way to terminate it > > is a kill -9. This seems almost like an OS level issue. We are opening > a > > ticket with Red Hat support to see what they say. > > > Thanks for the tip Coty. The logs show a normal start. The kill -3 output does not appear in the logs. I then did a kill -9 and that showed up. Logs: -- Logs begin at Wed 2020-07-01 12:37:09 PDT, end at Thu 2020-07-02 06:36:59 PDT. -- Jul 02 06:36:31 ecom-main.spokaneproduce.com systemd[1]: Started Apache Tomcat We b Application Container. -- Subject: Unit tomcat.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit tomcat.service has finished starting up. -- -- The start-up result is done. Jul 02 06:36:32 ecom-main.spokaneproduce.com server[26466]: Java virtual machine used: /usr/lib/jvm/jre/bin/java Jul 02 06:36:32 ecom-main.spokaneproduce.com server[26466]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar Jul 02 06:36:32 ecom-main.spokaneproduce.com server[26466]: main class used: org.apache.catalina.startup.Bootstrap Jul 02 06:36:32 ecom-main.spokaneproduce.com server[26466]: flags used: Jul 02 06:36:32 ecom-main.spokaneproduce.com server[26466]: options used: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Jul 02 06:36:32 ecom-main.spokaneproduce.com server[26466]: arguments used: start Jul 02 06:36:59 ecom-main.spokaneproduce.com systemd[1]: tomcat.service: main process exited, code=killed, status=9/KILL Jul 02 06:36:59 ecom-main.spokaneproduce.com systemd[1]: Unit tomcat.service entered failed state. Jul 02 06:36:59 ecom-main.spokaneproduce.com systemd[1]: tomcat.service failed.
Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
The RHEL 7 Tomcat package uses systemd and the journal to capture stdout instead of catalina.out. Did you check the journal to see if the thread dumps are logged there from your invocations of kill -3? You can use `journalctl -u tomcat` to check it. On Wed, Jul 1, 2020 at 6:58 PM Sean Neeley wrote: > On Wed, Jul 1, 2020 at 5:24 PM calder wrote: > > > On Wed, Jul 1, 2020, 15:32 Sean Neeley > wrote: > > > > > I tried switching from Java 1.8 to Java 11 to see if that makes a > > > difference. Now the VM Thread is using a lot less CPU: > > > > > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ > > COMMAND > > > 2320 tomcat20 0 4659072 47872 19904 R 99.9 0.6 22:15.16 java > > > 2326 tomcat20 0 4659072 47872 19904 R 4.6 0.6 0:56.43 VM > > > Thread > > > > > > I tried running jstack on the processes, but I get this: > > > > > > 2320: Unable to open socket file: target process not responding or > > HotSpot > > > VM not loaded > > > > > > > Did you attempt to run the command as the "Tomcat user"? > > > > BTW, Oracle recommends the use of "jcmd" over "jstack". Personally, I'd > > give Mission Control/Flight Recorder a go. > > > > I'm definitely running it as the tomcat user. I just tried jcmd with no > arguments and the command completely hangs. The only way to terminate it > is a kill -9. This seems almost like an OS level issue. We are opening a > ticket with Red Hat support to see what they say. >
Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
On Wed, Jul 1, 2020 at 5:24 PM calder wrote: > On Wed, Jul 1, 2020, 15:32 Sean Neeley wrote: > > > I tried switching from Java 1.8 to Java 11 to see if that makes a > > difference. Now the VM Thread is using a lot less CPU: > > > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ > COMMAND > > 2320 tomcat20 0 4659072 47872 19904 R 99.9 0.6 22:15.16 java > > 2326 tomcat20 0 4659072 47872 19904 R 4.6 0.6 0:56.43 VM > > Thread > > > > I tried running jstack on the processes, but I get this: > > > > 2320: Unable to open socket file: target process not responding or > HotSpot > > VM not loaded > > > > Did you attempt to run the command as the "Tomcat user"? > > BTW, Oracle recommends the use of "jcmd" over "jstack". Personally, I'd > give Mission Control/Flight Recorder a go. > I'm definitely running it as the tomcat user. I just tried jcmd with no arguments and the command completely hangs. The only way to terminate it is a kill -9. This seems almost like an OS level issue. We are opening a ticket with Red Hat support to see what they say.
Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
On Wed, Jul 1, 2020, 15:32 Sean Neeley wrote: > I tried switching from Java 1.8 to Java 11 to see if that makes a > difference. Now the VM Thread is using a lot less CPU: > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 2320 tomcat20 0 4659072 47872 19904 R 99.9 0.6 22:15.16 java > 2326 tomcat20 0 4659072 47872 19904 R 4.6 0.6 0:56.43 VM > Thread > > I tried running jstack on the processes, but I get this: > > 2320: Unable to open socket file: target process not responding or HotSpot > VM not loaded > Did you attempt to run the command as the "Tomcat user"? BTW, Oracle recommends the use of "jcmd" over "jstack". Personally, I'd give Mission Control/Flight Recorder a go.
Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
I tried switching from Java 1.8 to Java 11 to see if that makes a
difference. Now the VM Thread is using a lot less CPU:
PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
2320 tomcat20 0 4659072 47872 19904 R 99.9 0.6 22:15.16 java
2326 tomcat20 0 4659072 47872 19904 R 4.6 0.6 0:56.43 VM Thread
I tried running jstack on the processes, but I get this:
2320: Unable to open socket file: target process not responding or HotSpot
VM not loaded
I tried running strace on the process and see this over and over in the
output:
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100},
[pid 2334] <... futex resumed>)= -1 ETIMEDOUT (Connection timed
out)
[pid 2334] futex(0x3fff7c1fd228, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 2334] futex(0x3fff7c1fd254, FUTEX_WAIT_BITSET_PRIVATE, 17307,
{tv_sec=2074, tv_nsec=141673818}, 0x
[pid 2326] <... nanosleep resumed>NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
[pid 2326] nanosleep({tv_sec=0, tv_nsec=100}, NULL) = 0
/var/log/tomcat logs are still empty. I'm running out of ideas.
Sean
On Wed, Jul 1, 2020 at 12:55 PM wrote:
> Sean,
>
>
> > -Original Message-
> > From: Sean Neeley
> > Sent: Wednesday, July 01, 2020 12:26 PM
> > To: Tomcat Users List
> > Subject: Re: Having trouble with tomcat 7 installation on RHEL 7.8 power
> pc
> >
> > John, The top two processes are:
> >
> > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+
> COMMAND
> > 19360 tomcat20 0 3302144 42560 18944 R 99.9 0.5 46:02.37 java
> > 19363 tomcat20 0 3302144 42560 18944 R 98.4 0.5 45:48.50 VM
> Thread
> >
> > I tried kill -3 on both of them, plus the java process ID I see from a
> "ps"
> > command, several seconds apart. The only logs created are these, and
> they
> > are still all empty after the kill -3:
> > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 catalina.2020-07-01.log
> > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22
> host-manager.2020-07-01.log
> > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 localhost.2020-07-01.log
> > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 manager.2020-07-01.log
> >
> > Any other ideas?
> >
> > On Wed, Jul 1, 2020 at 12:09 PM
> > wrote:
> >
> > > Sean,
> > >
> > > > -Original Message-
> > > > From: Sean Neeley
> > > > Sent: Wednesday, July 01, 2020 11:15 AM
> > > > To: users@tomcat.apache.org
> > > > Subject: Having trouble with tomcat 7 installation on RHEL 7.8 power
> > > > pc
> > > >
> > > > I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8,
> > > power pc
> > > > system. As soon as the service starts, the java process uses 100%
> cpu.
> > > > Logs get created in /var/log/tomcat, but they all have size 0 bytes.
> > > > I
> > > have not
> > > > modified the standard configuration (tomcat.conf, server.xml, etc).
> > > > The tomcat packages that are installed are:
> > > >
> > > > tomcat-7.0.76-12.el7_8.noarch
> > > > tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch
> > > > tomcat-el-2.2-api-7.0.76-12.el7_8.noarch
> > > > tomcat-lib-7.0.76-12.el7_8.noarch
> > > > tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch
> > > >
> > > > Are there any tricks I can use to troubleshoot what is going wrong?
> > > > Without any logs being created, it seems impossible to determine the
> > > > problem. Thanks for helping.
> > > >
> > > > Sean
> > >
> > > Use kill -3 to take several thread dumps about 5-10 seconds apart.
> > > They will probably end up in catalina.out.
> > >
> > > You can also use top -H to see what individual threads are using CPU.
> > >
> > >
>
> Sorry, I don't know much more.
>
> It's interesting that the CPU is being consumed by "VM Thread." It
> doesn't look like an ordinary Tomcat or application problem. When I
> googled "java vm thread," I got some interesting results, such as this one:
>
>
> https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/hangloop.html
>
> That also describes the scenario where a thread dump doesn't seem to do
> anything.
>
>
>
RE: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
Sean, > -Original Message- > From: Sean Neeley > Sent: Wednesday, July 01, 2020 12:26 PM > To: Tomcat Users List > Subject: Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc > > John, The top two processes are: > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 19360 tomcat20 0 3302144 42560 18944 R 99.9 0.5 46:02.37 java > 19363 tomcat20 0 3302144 42560 18944 R 98.4 0.5 45:48.50 VM Thread > > I tried kill -3 on both of them, plus the java process ID I see from a "ps" > command, several seconds apart. The only logs created are these, and they > are still all empty after the kill -3: > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 catalina.2020-07-01.log > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 host-manager.2020-07-01.log > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 localhost.2020-07-01.log > -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 manager.2020-07-01.log > > Any other ideas? > > On Wed, Jul 1, 2020 at 12:09 PM > wrote: > > > Sean, > > > > > -Original Message- > > > From: Sean Neeley > > > Sent: Wednesday, July 01, 2020 11:15 AM > > > To: users@tomcat.apache.org > > > Subject: Having trouble with tomcat 7 installation on RHEL 7.8 power > > > pc > > > > > > I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8, > > power pc > > > system. As soon as the service starts, the java process uses 100% cpu. > > > Logs get created in /var/log/tomcat, but they all have size 0 bytes. > > > I > > have not > > > modified the standard configuration (tomcat.conf, server.xml, etc). > > > The tomcat packages that are installed are: > > > > > > tomcat-7.0.76-12.el7_8.noarch > > > tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch > > > tomcat-el-2.2-api-7.0.76-12.el7_8.noarch > > > tomcat-lib-7.0.76-12.el7_8.noarch > > > tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch > > > > > > Are there any tricks I can use to troubleshoot what is going wrong? > > > Without any logs being created, it seems impossible to determine the > > > problem. Thanks for helping. > > > > > > Sean > > > > Use kill -3 to take several thread dumps about 5-10 seconds apart. > > They will probably end up in catalina.out. > > > > You can also use top -H to see what individual threads are using CPU. > > > > Sorry, I don't know much more. It's interesting that the CPU is being consumed by "VM Thread." It doesn't look like an ordinary Tomcat or application problem. When I googled "java vm thread," I got some interesting results, such as this one: https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/hangloop.html That also describes the scenario where a thread dump doesn't seem to do anything.
Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
John, The top two processes are: PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 19360 tomcat20 0 3302144 42560 18944 R 99.9 0.5 46:02.37 java 19363 tomcat20 0 3302144 42560 18944 R 98.4 0.5 45:48.50 VM Thread I tried kill -3 on both of them, plus the java process ID I see from a "ps" command, several seconds apart. The only logs created are these, and they are still all empty after the kill -3: -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 catalina.2020-07-01.log -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 host-manager.2020-07-01.log -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 localhost.2020-07-01.log -rw-r--r-- 1 tomcat tomcat0 Jul 1 10:22 manager.2020-07-01.log Any other ideas? On Wed, Jul 1, 2020 at 12:09 PM wrote: > Sean, > > > -Original Message- > > From: Sean Neeley > > Sent: Wednesday, July 01, 2020 11:15 AM > > To: users@tomcat.apache.org > > Subject: Having trouble with tomcat 7 installation on RHEL 7.8 power pc > > > > I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8, > power pc > > system. As soon as the service starts, the java process uses 100% cpu. > > Logs get created in /var/log/tomcat, but they all have size 0 bytes. I > have not > > modified the standard configuration (tomcat.conf, server.xml, etc). The > > tomcat packages that are installed are: > > > > tomcat-7.0.76-12.el7_8.noarch > > tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch > > tomcat-el-2.2-api-7.0.76-12.el7_8.noarch > > tomcat-lib-7.0.76-12.el7_8.noarch > > tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch > > > > Are there any tricks I can use to troubleshoot what is going wrong? > > Without any logs being created, it seems impossible to determine the > > problem. Thanks for helping. > > > > Sean > > Use kill -3 to take several thread dumps about 5-10 seconds apart. They > will probably end up in catalina.out. > > You can also use top -H to see what individual threads are using CPU. > >
Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
Permissions of /var/log/tomcat look fine: drwxrwx---. 2 tomcat root 4096 Jul 1 08:34 tomcat And it is creating the logs, they are just empty: -rw-r--r-- 1 tomcat tomcat0 Jul 1 08:34 catalina.2020-07-01.log -rw-r--r-- 1 tomcat tomcat0 Jul 1 08:34 host-manager.2020-07-01.log -rw-r--r-- 1 tomcat tomcat0 Jul 1 08:34 localhost.2020-07-01.log -rw-r--r-- 1 tomcat tomcat0 Jul 1 08:34 manager.2020-07-01.log /var/log/messages shows: Jul 1 09:31:03 ecom-main systemd: Started Apache Tomcat Web Application Container. Jul 1 09:31:03 ecom-main server: Java virtual machine used: /usr/lib/jvm/jre/bin/java Jul 1 09:31:03 ecom-main server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar Jul 1 09:31:03 ecom-main server: main class used: org.apache.catalina.startup.Bootstrap Jul 1 09:31:03 ecom-main server: flags used: Jul 1 09:31:03 ecom-main server: options used: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager Jul 1 09:31:03 ecom-main server: arguments used: start On Wed, Jul 1, 2020 at 12:00 PM calder wrote: > On Wed, Jul 1, 2020, 11:15 Sean Neeley wrote: > > > I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8, power > > pc system. As soon as the service starts, the java process uses 100% > cpu. > > Logs get created in /var/log/tomcat, but they all have size 0 bytes. I > > have not modified the standard configuration (tomcat.conf, server.xml, > > etc). The tomcat packages that are installed are: > > > > tomcat-7.0.76-12.el7_8.noarch > > tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch > > tomcat-el-2.2-api-7.0.76-12.el7_8.noarch > > tomcat-lib-7.0.76-12.el7_8.noarch > > tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch > > > > Are there any tricks I can use to troubleshoot what is going wrong? > > > Check "syslog" for clues. > > Also, double-check the permissions for /var/log/tomcat. >
RE: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
Sean, > -Original Message- > From: Sean Neeley > Sent: Wednesday, July 01, 2020 11:15 AM > To: users@tomcat.apache.org > Subject: Having trouble with tomcat 7 installation on RHEL 7.8 power pc > > I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8, power pc > system. As soon as the service starts, the java process uses 100% cpu. > Logs get created in /var/log/tomcat, but they all have size 0 bytes. I have > not > modified the standard configuration (tomcat.conf, server.xml, etc). The > tomcat packages that are installed are: > > tomcat-7.0.76-12.el7_8.noarch > tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch > tomcat-el-2.2-api-7.0.76-12.el7_8.noarch > tomcat-lib-7.0.76-12.el7_8.noarch > tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch > > Are there any tricks I can use to troubleshoot what is going wrong? > Without any logs being created, it seems impossible to determine the > problem. Thanks for helping. > > Sean Use kill -3 to take several thread dumps about 5-10 seconds apart. They will probably end up in catalina.out. You can also use top -H to see what individual threads are using CPU.
Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
On Wed, Jul 1, 2020, 11:15 Sean Neeley wrote: > I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8, power > pc system. As soon as the service starts, the java process uses 100% cpu. > Logs get created in /var/log/tomcat, but they all have size 0 bytes. I > have not modified the standard configuration (tomcat.conf, server.xml, > etc). The tomcat packages that are installed are: > > tomcat-7.0.76-12.el7_8.noarch > tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch > tomcat-el-2.2-api-7.0.76-12.el7_8.noarch > tomcat-lib-7.0.76-12.el7_8.noarch > tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch > > Are there any tricks I can use to troubleshoot what is going wrong? Check "syslog" for clues. Also, double-check the permissions for /var/log/tomcat.
