Chris, Ok, I will investigate nginx side as well. Thank you for the pointers
On Wed, 24 Jun 2020, 19:45 Christopher Schultz, < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > > Ayub, > > On 6/24/20 11:05, Ayub Khan wrote: > > If some open file is owned by nginx why would it show up if I run > > the below command> sudo lsof -p $(cat /var/run/tomcat8.pid) > > Because network connections have two ends. > > - -chris > > > On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Ayub, > > > > On 6/23/20 19:17, Ayub Khan wrote: > >>>> Yes we have nginx as reverse proxy, below is the nginx > >>>> config. We notice this issue only when there is high number > >>>> of requests, during non peak hours we do not see this issue.> > >>>> location /myapp/myservice{ #local machine proxy_pass > >>>> http://localhost:8080; proxy_http_version 1.1; > >>>> > >>>> proxy_set_header Connection $connection_upgrade; > >>>> proxy_set_header Upgrade $http_upgrade; > >>>> proxy_set_header Host $host; > >>>> proxy_set_header X-Real-IP $remote_addr; > >>>> proxy_set_header X-Forwarded-For > >>>> $proxy_add_x_forwarded_for; > >>>> > >>>> > >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; } > > > > You might want to read about tuning nginx to drop connections after > > a certain period of time, number of requests, etc. Looks like > > either a bug in nginx or a misconfiguration which allows > > connections to stick-around like this. You may have to ask the > > nginx people. I have no experience with nginx myself, while others > > here may have some experience. > > > >>>> location / { # if using AWS Load balancer, this bit checks > >>>> for the presence of the https proto flag. if regular http is > >>>> found, then issue a redirect > > to hit > >>>> the https endpoint instead if ($http_x_forwarded_proto != > >>>> 'https') { rewrite ^ https://$host$request_uri? permanent; } > >>>> > >>>> proxy_pass http://127.0.0.1:8080; > >>>> proxy_http_version 1.1; > >>>> > >>>> proxy_set_header Connection $connection_upgrade; > >>>> proxy_set_header Upgrade $http_upgrade; > >>>> proxy_set_header Host $host; > >>>> proxy_set_header X-Real-IP $remote_addr; > >>>> proxy_set_header X-Forwarded-For > >>>> $proxy_add_x_forwarded_for; > >>>> > >>>> > >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; } > >>>> > >>>> *below is the connector* > >>>> > >>>> <Connector port="8080" > >>>> protocol="org.apache.coyote.http11.Http11NioProtocol" > >>>> connectionTimeout="2000" maxThreads="50000" > >>>> URIEncoding="UTF-8" redirectPort="8443" /> > > > > 50k threads is a LOT of threads. Do you expect to handle 50k > > requests simultaneously? > > > >>>> these ports are random, I am not sure who owns the process. > >>>> > >>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port > >>>> 55866 is a random port. > > I'm sure you'll find that 55866 is owned by nginx. netstat will > > tell you . > > > > I think you need to look at your nginx configuration. It would also > > be a great time to upgrade to a supported version of Tomcat. I > > would recommend 8.5.56 or 9.0.36. > > > > -chris > > > >>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz < > >>>> ch...@christopherschultz.net> wrote: > >>>> > >>>> Ayub, > >>>> > >>>> On 6/23/20 16:23, Ayub Khan wrote: > >>>>>>> I executed *sudo lsof -p $(cat /var/run/tomcat8.pid) > >>>>>>> *and I saw the below output, some in CLOSE_WAIT and > >>>>>>> others in ESTABLISHED. If there are 200 open file > >>>>>>> descriptors 160 are in CLOSE_WAIT state. When the count > >>>>>>> for CLOSE_WAIT increases I just have to restart > >>>>>>> tomcat. > >>>>>>> > >>>>>>> java 65189 tomcat8 715u IPv6 > >>>>>>> 237878311 0t0 TCP localhost:http-alt->localhost:43760 > >>>>>>> (CLOSE_WAIT) java 65189 tomcat8 716u IPv6 > >>>>>>> 237848923 0t0 TCP > >>>>>>> localhost:http-alt->localhost:40568 (CLOSE_WAIT) > >>>> > >>>> These are connections from some process into Tomcat listening > >>>> on port 8080 (that's what localhost:http-alt is). So what > >>>> process owns the outgoing connection on port 40568 on the > >>>> same host? > >>>> > >>>> Are you using a reverse proxy? > >>>> > >>>>>>> most of the open files are in CLOSE_WAIT state I do not > >>>>>>> see anything related to database ip. > >>>> > >>>> Agreed. It looks like you have a reverse proxy who is > >>>> losing-track of connections, or who is (re)opening > >>>> connections when it may be unnecessar y. > >>>> > >>>> Can you share your <Connector> configuration from > >>>> server.xml? Remember to remove any secrets. > >>>> > >>>> -chris > >>>> > >>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher < > >>>>>>> felix.schumac...@internetallee.de> wrote: > >>>>>>> > >>>>>>>> > >>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan: > >>>>>>>>> Felix, > >>>>>>>>> > >>>>>>>>> I executed ls -l /proc/$(cat > >>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the > >>>>>>>> output > >>>>>>>>> I see majority of them are related to sockets as > >>>>>>>>> shown below, some of > >>>>>>>> them > >>>>>>>>> point to the jar file of tomcat and others to the > >>>>>>>>> log file which is > >>>>>>>> created. > >>>>>>>>> > >>>>>>>>> socket:[2084570754] socket:[2084579487] > >>>>>>>>> socket:[2084578478] socket:[2084570167] > >>>>>>>> > >>>>>>>> Can you try the other command (lsof -p $(cat > >>>>>>>> ...tomcat.pid))? It should give a bit more details on > >>>>>>>> the used sockets that the proc directory. > >>>>>>>> > >>>>>>>> Felix > >>>>>>>> > >>>>>>>>> > >>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher < > >>>>>>>>> felix.schumac...@internetallee.de> wrote: > >>>>>>>>> > >>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan: > >>>>>>>>>>> Chris, > >>>>>>>>>>> > >>>>>>>>>>> I am using HikariCP for connection pooling. If > >>>>>>>>>>> the database is leaking connections then I > >>>>>>>>>>> should see connection not available exception. > >>>>>>>>>>> > >>>>>>>>>>> How do I find out which file descriptors are > >>>>>>>>>>> leaking ? these are not > >>>>>>>>>> files > >>>>>>>>>>> open on disk as there is no explicit disk file > >>>>>>>>>>> I/O in this application. > >>>>>>>>>>> > >>>>>>>>>>> I just use the below command to check for open > >>>>>>>>>>> file descriptors: > >>>>>>>>>>> > >>>>>>>>>>> watch "sudo ls /proc/`cat > >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l" > >>>>>>>>>> You could have a look at the name of the files in > >>>>>>>>>> the pids proc > >>>>>>>> directory. > >>>>>>>>>> > >>>>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ > >>>>>>>>>> > >>>>>>>>>> Or you could use the tool lsof to find the open > >>>>>>>>>> file descriptors. > >>>>>>>>>> > >>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid) > >>>>>>>>>> > >>>>>>>>>> For both calls you should first change to the uid > >>>>>>>>>> of the tomcat user or use sudo as in your > >>>>>>>>>> example. > >>>>>>>>>> > >>>>>>>>>> Felix > >>>>>>>>>> > >>>>>>>>>>> Thanks and Regards Ayub > >>>>>>>>>>> > >>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher > >>>>>>>>>>> Schultz < ch...@christopherschultz.net> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Ayub, > >>>>>>>>>>> > >>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote: > >>>>>>>>>>>>>> Sorry we are using 8.0.32 version of > >>>>>>>>>>>>>> tomcat. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> below is the configuration: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Server version: Apache Tomcat/8.0.32 > >>>>>>>>>>>>>> (Ubuntu) Server built: Jan 24 2020 > >>>>>>>>>>>>>> 16:24:30 UTC Server number: 8.0.32.0 OS > >>>>>>>>>>>>>> Name: Linux OS Version: 4.4.0-1087-aws > >>>>>>>>>>>>>> Architecture: amd64 JVM Version: > >>>>>>>>>>>>>> 1.8.0_181-b13 JVM Vendor: Oracle > >>>>>>>>>>>>>> Corporation > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I use the below command to check the > >>>>>>>>>>>>>> file descriptors: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> watch "sudo ls /proc/`cat > >>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l" > >>>>>>>>>>> So you know there is some kind of increase in > >>>>>>>>>>> file-handle use, but you don't know what types > >>>>>>>>>>> of file handles are increasing, right? > >>>>>>>>>>> > >>>>>>>>>>> Can you try to find out which kinds of file > >>>>>>>>>>> handles are increasing? > >>>>>>>>>>> > >>>>>>>>>>> I have a sneaking suspicion that it's your > >>>>>>>>>>> database connections and not actually files > >>>>>>>>>>> open on the disk. > >>>>>>>>>>> > >>>>>>>>>>> Are you using a database connection pool? If > >>>>>>>>>>> not, you should really use one and limit the > >>>>>>>>>>> number of connections to something sane. If you > >>>>>>>>>>> are using one, are you monitoring it to see how > >>>>>>>>>>> many connections are actually being used? Are > >>>>>>>>>>> you sure you are using proper resource > >>>>>>>>>>> management[1]? Even a single code-path that > >>>>>>>>>>> leaks connections can leak them quickly under > >>>>>>>>>>> load. > >>>>>>>>>>> > >>>>>>>>>>>>>> When there an issue related to broken > >>>>>>>>>>>>>> files, this value keeps increasing, the > >>>>>>>>>>>>>> only way to bring it down is to remove vm > >>>>>>>>>>>>>> instance from AWS load balancer.> Which > >>>>>>>>>>>>>> version of tomcat should I install ? > >>>>>>>>>>> Tomcat 8.0.x hasn't been supported since its > >>>>>>>>>>> last release on 29 June 2018. That was 8.0.53. > >>>>>>>>>>> Your release is from 8 February 2016 and is > >>>>>>>>>>> dangerously out of date (unless you are using > >>>>>>>>>>> the Ubuntu-packaged version, in which case I > >>>>>>>>>>> hope they kept-up with security patches thee > >>>>>>>>>>> past 4 years). > >>>>>>>>>>> > >>>>>>>>>>> -chris > >>>>>>>>>>> > >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM > >>>>>>>>>>>>>> Christopher Schultz < > >>>>>>>>>>>>>> ch...@christopherschultz.net> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Ayub, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote: > >>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases > >>>>>>>>>>>>>>>>> open files on ubuntu AWS > >>>>>>>>>>>>>> Which exact version of Tomcat 8.5? If > >>>>>>>>>>>>>> you aren't running the latest version > >>>>>>>>>>>>>> (8.5.56), please upgrade and re-test. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> If there is slow response from db I > >>>>>>>>>>>>>>>>> see this stack trace and the open > >>>>>>>>>>>>>>>>> files goes high and the only way to > >>>>>>>>>>>>>>>>> open files go down is to remove the > >>>>>>>>>>>>>>>>> instance from Amazon load > >>>>>>>>>>>>>>>>> balancer. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Is there a way to keep the open > >>>>>>>>>>>>>>>>> files low even when Broken pipe > >>>>>>>>>>>>>>>>> error is thrown ? > >>>>>>>>>>>>>> What is your evidence that file handles > >>>>>>>>>>>>>> are being left open? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Which file handles are being left open? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -chris > >>>>>>>>>>>>>>> > >>>>>>>>>> ------------------------------------------------------------- > - --- > > > >>>>>>>>>> > - --- > >>>>> > >>>>> > >>>>>>>>>> > > --------------------------------------------------------------------- > >>>>> > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For > >>>>> additional commands, e-mail: users-h...@tomcat.apache.org > >>>>> > >>>>> > >>>> > >> > >> --------------------------------------------------------------------- > >> > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zgOIACgkQHPApP6U8 > pFhnWQ//SBTsCR/g8cDgrC+ju6x3BA8kpEmNbA4BP5fY8I7nrDoiegsrZDbRsoWI > rNlH3i3pLlonTbbLQN/6AeWItmgM88bupy007cEcNrNivztkOPscmFOmPZP7lNaD > JfgXQvTF1q8LFL+R120GGVR+OKM06dZ4rYbWbu+Ju46lYX6q2iVlb+n8XmmPBIPg > 9dEE4kXNc+A3ZI+3JxLoWnxnDSxpG2Od2WDwt+dLV0E7XkHJCMKGqtzbPWcUtaUR > nh9rywiO4ODk57VbtU426yB6OrNxbwj2OOKpETKVfLpUsdwzvgEZgzbfeUqPpR4p > OpIXxjgBeGFhk+ssafvqOMIMI4EO/m/+uttyfQhAh05ozP99NLTZXbDcnY9gFEeY > Rk7Z7qnGey3erk83dfR3X56zovuiS6k6vHKLZcp6Js5ta1cQa9D8hOjxcVRaDWlz > AeKNEiwuuIo2tfsl2OCAxkoNpl+dooE2aXcvAWbrI7pGXMWwANzEGw4NmIUTOvpS > tjR4OqaOKiRo1TN66JZVeol0ZFlaz9InxHTEasHrdn80YJGMIFL7UmR6koZGUE2G > 7k0yukOUQCDLCPc3HwCCfOw/rkygLtxKShMEcZDp2gH3AZo0z1AKBDGWyi7XyMKa > MO4DkMHhp28aRZ84Dtsyvxh/HuxBt9Vl2hj/oOkBMVbfrIT+3bg= > =q6xp > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >