Yeggy,
On 5/19/21 19:37, Yeggy Javadi wrote:
Hi Chris,
Thanks for your email. Below are replies to your questions:
1. Did you upgrade anything recently (like Java VM)?
[YJ] To support Linux 8, only Postgres was upgraded from version 9.3 to 9.6.
2. What is error you are seeing? A full stack trace would be helpful.
[YJ] The application error can occur in any place with the (Too many open
files) error when the limit of open files for tomcat has reached (262144 in my
environment), for example, I can get a connectivity error when pulling info
from a server as below :
[InventoryPullerTask - 10.1.6.25] ERROR FSSDataCollectorService - Error : The
XML configuration file failed to be retrieved for server 10.1.6.25. Check
server connectivity.Error retrieving IPStorConfg for server=10.1.6.25
error=java.io.FileNotFoundException:
/usr/local/apache-tomcat-8.5.59/webapps/ROOT/WEB-INF/lib/spring-orm-3.2.10.RELEASE.jar
(Too many open files) restatus=-1 output=
3. What is your <Connector> configuration?
[YJ] It is as below in server.xml:
<Connector address="0.0.0.0" port="80" protocol="HTTP/1.1"
maxThreads="150"
minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true"
compression="on"
compressionMinSize="2048"
noCompressionUserAgents="gozilla, traviata"
compressableMimeType="text/html,text/xml,application/json"
redirectPort="443" />
...........
<Connector address="0.0.0.0" port="443"
protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
clientAuth="false" sslEnabledProtocols="TLSv1.2"
ciphers="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA"
keystoreFile="conf/freestor" keystorePass="freestor"/>
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector address="0.0.0.0" port="8009" protocol="AJP/1.3" redirectPort="443"
secretRequired="false" />
You have lots of settings on the HTTP connector but not the HTTPS one.
Is that intentional? Which one are you using? Or both?
Are you using AJP? If not, disable the connetor entirely. Definitely
never enable it on 0.0.0.0 unless you really know what you are doing.
4. Are you counting all the open files for a single process or all of the
sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
[YJ] I am just getting the process ID of tomcat and count open files for that
process
Ok.
5. Running lsof, netstat, etc. can you see if some large number of those
sockets are bound to any specific port (listen or connect)?
[YJ] Here is the netstat output:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 Yeggy-F8-FMSVA:ssh 10.12.3.33:55236 ESTABLISHED
tcp 0 64 Yeggy-F8-FMSVA:ssh 10.197.255.10:60378 ESTABLISHED
tcp 0 0 localhost.loca:postgres localhost.localdo:36846 ESTABLISHED
That's not helpful. I want to see what the JVM process is holding, not
what the whole OS is handling.
My guess is that lsof is going to be a better bet for inspecting the
process. But if you get "too many open files" being thrown inside the
JVM and you don't see a 1/4 million lines of "lsof" output on your
process, then something else is going on. But remember to limit your
lsof to *just the JVM* process.
-chris
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Monday, May 17, 2021 5:18 PM
To: users@tomcat.apache.org
Subject: Re: apache-tomcat-8.5.59 too many open files on Linux 8
Yeggy,
On 5/17/21 15:17, Yeggy Javadi wrote:
Hi,
I am seeing too many TCP sockets open by tomcat on Oracle Linux 8:
java 51798 root 6601u sock 0,9 0t0 3959342 protocol:
TCPv6
I was running the same application on Linux 7.6 with the same
apache-tomcat version without seeing this issue. I saw that RedHat has
removed apache-tomcat fom their Linux 8 distribution. Please help with
throubleshooting and indicating if there is any changes in Linux 8
that can explain this
Configuration:
apache-tomcat-8.5.59
OEL Linux 8 kernel 4.18.0-193.28.1.el8_2.x86_64 Web-based application
using Postgres and elastic search databases
# ps -ef | grep tomcat
root 51798 1 1 11:16 ? 00:06:01 /usr/local/jre/bin/java
-Djava.util.logging.config.file=/usr/local/apache-tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -d64 -server
-Xms1800m -Xmx8192m -XX:MaxMetaspaceSize=1800m
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
-Dignore.endorsed.dirs= -classpath
/usr/local/apache-tomcat/bin/bootstrap.jar:/usr/local/apache-tomcat/bin/tomcat-juli.jar
-Dcatalina.base=/usr/local/apache-tomcat
-Dcatalina.home=/usr/local/apache-tomcat
-Djava.io.tmpdir=/usr/local/apache-tomcat/temp
org.apache.catalina.startup.Bootstrap start
root 54122 54037 0 19:43 pts/1 00:00:00 grep --color=auto tomcat
# cat /proc/51798/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 256518 256518 processes
Max open files 262144 262144 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256518 256518 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Too may open files:
# lsof -p 51798 | wc -l
6679
Most of open files are TCP sockets:
java 51798 root 6601u sock 0,9 0t0 3959342 protocol:
TCPv6
# lsof -p 51798 | grep "protocol: TCPv6" | wc -l
6413
Some potentially dumb questions:
1. Did you upgrade anything recently (like Java VM)?
2. What is error you are seeing? A full stack trace would be helpful.
3. What is your <Connector> configuration?
4. Are you counting all the open files for a single process or all of the
sub-processes which represent the threads of the main process?
Different kernel versions count things differently.
5. Running lsof, netstat, etc. can you see if some large number of those
sockets are bound to any specific port (listen or connect)?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
