RE: apache-tomcat-8.5.59 too many open files on Linux 8
Hi, Here its is: # lsof -p 130244 | grep "protocol: TCPv6"| wc -l 19948 # netstat -p -a -6 --tcp | grep 130244 tcp6 0 0 [::]:pcsync-https [::]:* LISTEN 130244/java tcp6 0 0 [::]:https [::]:* LISTEN 130244/java tcp6 0 0 [::]:37537 [::]:* LISTEN 130244/java tcp6 0 0 localhost.localdoma:mxi [::]:* LISTEN 130244/java tcp6 0 0 [::]:8009 [::]:* LISTEN 130244/java tcp6 0 0 [::]:rmiregistry[::]:* LISTEN 130244/java tcp6 0 0 [::]:http [::]:* LISTEN 130244/java tcp6 86 0 Yeggy-F8-FMSVA:39680172.22.22.192:https CLOSE_WAIT 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54772Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 localhost.localdo:42664 localhost.loca:postgres ESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54782Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54766Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 localhost.localdo:42662 localhost.loca:postgres ESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54778Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 localhost.localdo:46966 localhost.localdo:11753 ESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54788Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54770Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64799 ESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54790Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54776Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54786Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54780Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 localhost.localdo:45736 localhost.loca:postgres ESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54768Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54784Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 localhost.localdo:42660 localhost.loca:postgres ESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:54774Yeggy-F8-FMSVA:vraceESTABLISHED 130244/java tcp6 0 0 localhost.localdo:45734 localhost.loca:postgres ESTABLISHED 130244/java tcp6 0 0 Yeggy-F8-FMSVA:http 10.197.255.10:64798 ESTABLISHED 130244/java tcp6 0 0 localhost.localdo:41016 localhost.localdo:vrace ESTABLISHED 130244/java Thanks -Original Message- From: Noelette Stout Sent: Friday, May 21, 2021 8:28 AM To: Tomcat Users List Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 ss has all the same options as netstat On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) wrote: > Soyrry to top-post, but it's getting cluttered down there.. > > The next thing that you may want to do : > > > netstat -p -a -6 --tcp > > That is an alternative list of sockets, which also shows the "tcp state" > of the sockets. > To get only the ones of the tomcat JVM PID, filter with grep based of > the last column. > The type of thing you are looking for is the column which should show > "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc.. > > The options above : > -p : show PID and program > -a : show all sockets states > -6 : only inet v6 > --tcp : only TCP sockets > > "netstat" may not be on your system by default, and you may need to > install it. > An alternative is "ss", but I don't know the options. > > > On 21.05.2021 02:14, Yeggy Javadi wrote: > > Hi, > > Yes; that is what I get and as you can see among 8028 open files, > > 7474 > are for TCPv6 sockets: > > java130244 root 7805u sock0,9 0t0 12294251 > protocol: TCPv6 > > > > # ps -ef | grep tomcat > > root 130244 1 1 11:01 ?00:06:20 > /usr/local/jre/bin/java > -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging. > properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 > -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dignore.endorsed.dirs= -classpath > /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bi > n/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat > -Dcatalina.home=/usr/local/apache-tomcat > -Djava.io.tmpdir=/usr/local/apache-tomcat/temp > org.apache.catalina.startup.Bootstrap start > > root 132566 132492
Re: apache-tomcat-8.5.59 too many open files on Linux 8
ss has all the same options as netstat On Fri, May 21, 2021 at 3:51 AM André Warnier (tomcat/perl) wrote: > Soyrry to top-post, but it's getting cluttered down there.. > > The next thing that you may want to do : > > > netstat -p -a -6 --tcp > > That is an alternative list of sockets, which also shows the "tcp state" > of the sockets. > To get only the ones of the tomcat JVM PID, filter with grep based of the > last column. > The type of thing you are looking for is the column which should show > "LISTEN", or > "ESTABLISHED" or "CLOSE_WAIT" etc.. > > The options above : > -p : show PID and program > -a : show all sockets states > -6 : only inet v6 > --tcp : only TCP sockets > > "netstat" may not be on your system by default, and you may need to > install it. > An alternative is "ss", but I don't know the options. > > > On 21.05.2021 02:14, Yeggy Javadi wrote: > > Hi, > > Yes; that is what I get and as you can see among 8028 open files, 7474 > are for TCPv6 sockets: > > java130244 root 7805u sock0,9 0t0 12294251 > protocol: TCPv6 > > > > # ps -ef | grep tomcat > > root 130244 1 1 11:01 ?00:06:20 > /usr/local/jre/bin/java > -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 > -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m > -Djdk.tls.ephemeralDHKeySize=2048 > -Djava.protocol.handler.pkgs=org.apache.catalina.webresources > -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 > -Dignore.endorsed.dirs= -classpath > /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar > -Dcatalina.base=/usr/local/apache-tomcat > -Dcatalina.home=/usr/local/apache-tomcat > -Djava.io.tmpdir=/usr/local/apache-tomcat/temp > org.apache.catalina.startup.Bootstrap start > > root 132566 132492 0 20:10 pts/100:00:00 grep --color=auto > tomcat > > > > #lsof -p 130244 | wc -l > > 8028 > > > > #lsof -p 130244 | grep "protocol: TCPv6"| wc -l > > 7474 > > > > Thanks > > > > -Original Message- > > From: André Warnier (tomcat/perl) > > Sent: Thursday, May 20, 2021 4:19 PM > > To: users@tomcat.apache.org > > Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 > > > > Hi. > > According to the list below, you have 2 java (JVM) processes running on > your system. > > One (PID = 130244) is the JVM which runs tomcat. This is visible when > you look at the whole command-line. > > > > The other (PID = 130516) runs ElasticSearch, which I believe is not > relevant here. > > > > So you should run lsof with the "-p 130244" option, to show only the > files opened by the tomcat JVM. > > To show the current Tomcat JVM PID, do e.g. : ps -ef | grep > "apache-tomcat" > > (or anything else unique in that line) > > > > > > On 20.05.2021 21:00, Yeggy Javadi wrote: > >> Hi Chris, > >> Please indicate how to show lsof or netstat to *just the JVM* process. > >> > >> Below is the list of running processes on my server: > >> UID PIDPPID C STIME TTY TIME CMD > >> root 1 0 0 May07 ?00:00:14 > /usr/lib/systemd/systemd --switched-root --system --deserialize 17 > >> root 2 0 0 May07 ?00:00:00 [kthreadd] > >> root 3 2 0 May07 ?00:00:00 [rcu_gp] > >> root 4 2 0 May07 ?00:00:00 [rcu_par_gp] > >> root 6 2 0 May07 ?00:00:00 > [kworker/0:0H-kblockd] > >> root 8 2 0 May07 ?00:00:00 [mm_percpu_wq] > >> root 9 2 0 May07 ?00:00:00 [ksoftirqd/0] > >> root 10 2 0 May07 ?00:02:18 [rcu_sched] > >> root 11 2 0 May07 ?00:00:00 [migration/0] > >> root 12 2 0 May07 ?00:00:00 [watchdog/0] > >> root 13 2 0 May07 ?00:00:00 [cpuhp/0] > >> root 14 2 0 May07 ?00:00:00 [cpuhp/1] > >> root 15 2 0 May07 ?00:00:00 [watchdog/1] > >> root 16 2 0 May07 ?00:00:00 [migration/1] > >> root 17 2 0 May07 ?00:00:00 [ksoftirqd/1] > >> root 19 2 0 May07 ?00:00:00 > [kworker/1:0H-kblockd] > >> root 20 2 0 May07 ?00:00:00 [cpuhp/2] > >> root 21 2 0 May07 ?00:00:00 [watchdog/2] > >> root 22 2 0 May07 ?00:00:00 [migration/2] > >> root 23 2 0 May07 ?00:00:00 [ksoftirqd/2] > >> root 25 2 0 May07 ?00:00:00 > [kworker/2:0H-kblockd] > >> root 26 2 0 May07 ?00:00:00 [cpuhp/3] > >> root 27 2 0 May07 ?00:00:00 [watchdog/3] > >> root 28 2 0 May07 ?00:00:00 [migration/3] > >> root 29 2 0 May07 ?00:00:00 [ksoftirqd/3] > >> root 31 2 0 May07 ?00:00:00 >
Re: apache-tomcat-8.5.59 too many open files on Linux 8
Soyrry to top-post, but it's getting cluttered down there.. The next thing that you may want to do : > netstat -p -a -6 --tcp That is an alternative list of sockets, which also shows the "tcp state" of the sockets. To get only the ones of the tomcat JVM PID, filter with grep based of the last column. The type of thing you are looking for is the column which should show "LISTEN", or "ESTABLISHED" or "CLOSE_WAIT" etc.. The options above : -p : show PID and program -a : show all sockets states -6 : only inet v6 --tcp : only TCP sockets "netstat" may not be on your system by default, and you may need to install it. An alternative is "ss", but I don't know the options. On 21.05.2021 02:14, Yeggy Javadi wrote: Hi, Yes; that is what I get and as you can see among 8028 open files, 7474 are for TCPv6 sockets: java130244 root 7805u sock0,9 0t0 12294251 protocol: TCPv6 # ps -ef | grep tomcat root 130244 1 1 11:01 ?00:06:20 /usr/local/jre/bin/java -Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server -Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/apache-tomcat -Dcatalina.home=/usr/local/apache-tomcat -Djava.io.tmpdir=/usr/local/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start root 132566 132492 0 20:10 pts/100:00:00 grep --color=auto tomcat #lsof -p 130244 | wc -l 8028 #lsof -p 130244 | grep "protocol: TCPv6"| wc -l 7474 Thanks -Original Message- From: André Warnier (tomcat/perl) Sent: Thursday, May 20, 2021 4:19 PM To: users@tomcat.apache.org Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8 Hi. According to the list below, you have 2 java (JVM) processes running on your system. One (PID = 130244) is the JVM which runs tomcat. This is visible when you look at the whole command-line. The other (PID = 130516) runs ElasticSearch, which I believe is not relevant here. So you should run lsof with the "-p 130244" option, to show only the files opened by the tomcat JVM. To show the current Tomcat JVM PID, do e.g. : ps -ef | grep "apache-tomcat" (or anything else unique in that line) On 20.05.2021 21:00, Yeggy Javadi wrote: Hi Chris, Please indicate how to show lsof or netstat to *just the JVM* process. Below is the list of running processes on my server: UID PIDPPID C STIME TTY TIME CMD root 1 0 0 May07 ?00:00:14 /usr/lib/systemd/systemd --switched-root --system --deserialize 17 root 2 0 0 May07 ?00:00:00 [kthreadd] root 3 2 0 May07 ?00:00:00 [rcu_gp] root 4 2 0 May07 ?00:00:00 [rcu_par_gp] root 6 2 0 May07 ?00:00:00 [kworker/0:0H-kblockd] root 8 2 0 May07 ?00:00:00 [mm_percpu_wq] root 9 2 0 May07 ?00:00:00 [ksoftirqd/0] root 10 2 0 May07 ?00:02:18 [rcu_sched] root 11 2 0 May07 ?00:00:00 [migration/0] root 12 2 0 May07 ?00:00:00 [watchdog/0] root 13 2 0 May07 ?00:00:00 [cpuhp/0] root 14 2 0 May07 ?00:00:00 [cpuhp/1] root 15 2 0 May07 ?00:00:00 [watchdog/1] root 16 2 0 May07 ?00:00:00 [migration/1] root 17 2 0 May07 ?00:00:00 [ksoftirqd/1] root 19 2 0 May07 ?00:00:00 [kworker/1:0H-kblockd] root 20 2 0 May07 ?00:00:00 [cpuhp/2] root 21 2 0 May07 ?00:00:00 [watchdog/2] root 22 2 0 May07 ?00:00:00 [migration/2] root 23 2 0 May07 ?00:00:00 [ksoftirqd/2] root 25 2 0 May07 ?00:00:00 [kworker/2:0H-kblockd] root 26 2 0 May07 ?00:00:00 [cpuhp/3] root 27 2 0 May07 ?00:00:00 [watchdog/3] root 28 2 0 May07 ?00:00:00 [migration/3] root 29 2 0 May07 ?00:00:00 [ksoftirqd/3] root 31 2 0 May07 ?00:00:00 [kworker/3:0H-kblockd] root 32 2 0 May07 ?00:00:00 [cpuhp/4] root 33 2 0 May07 ?00:00:00 [watchdog/4] root 34 2 0 May07 ?00:00:00 [migration/4] root 35 2 0 May07 ?00:00:00 [ksoftirqd/4] root 37 2 0 May07 ?00:00:00 [kworker/4:0H-kblockd] root 38 2 0 May07 ?00:00:00 [cpuhp/5] root 39 2 0 May07 ?
Re: ISAPI redirector for Microsoft IIS, Jboss EAP 7.2 - sticky session issue
On 21/05/2021 05:51, Mathiazhagan, Saravanakumar TPC wrote:
Hi Mark,
Thanks for the quick response.
I suspect some sort of configuration issue. A guess would be that JBoss
EAOP isn't configured to append the jvmRoute (to use the Tomcat
configuration setting name) to the session ID.
I configured IE to show prompt when cookies are set. I can see the jvmRoute
value in JSessionID generated, with the format {someID}.{workerName}.
You may need to look at the access log and/or network traffic to see
what is going on. That usually sheds some light on these sorts of issues.
I cross verified the access logs. One thing I noticed is consistently the
request switches between servers. So, every time the server gets request, a
new session (JSESSIONID) is being sent out.
From the network side too, no special rules are setup.
With same jvmRoute and network settings, sticky session works properly when I
use Jboss 6.4 version and 32-bit DLL.
If the 32 bit DLL has to work fine with Jboss7.2, then I get a feeling I am
missing something, but not able to figure out exactly what the issue is.
Please let me know if there are any particular configurations that need to be
checked.
At this point, I'd suggest tracing a series of requests from client to
server and checking, at each stage, that the behaviour is as you expect.
Where does it go wrong? That may point you in the right direction.
I'd be wanting to check things like:
- does a new request get a new session ID
- does the worker name match the node that handled the request
- do the access logs confirm the request went to the right worker
- for the second request, does the client sent the session cookie
- does the lb see the session cookie and route the request to the right
host
It sounds like things are going wrong around this point.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
