Chris,
What do you suggest now to debug this issue ? Check with Nginx support if
they can verify it ?
On Thu, Jun 25, 2020 at 8:17 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Ayub,
>
> On 6/25/20 11:06, Ayub Khan wrote:
> > Was just thinking if the file descriptors belonged to nginx why do
> > they disappear as soon as I restart tomcat ? I tried restarting
> > nginx and the open file descriptors don't disappear.
>
> When you restart Tomcat, the OS cleans-up the TCP/IP stack. Tomcat is
> waiting on some cleanup information on those sockets. nginx has
> evidently given-up on them and so the OS has adopted them.
>
> > When I execute lsof -p <nginx pid> I do not see file descriptors
> > in close wait state
>
> Because nginx has cleaned them up already.
>
> I could encourage you to take a look at the TCP/IP state diagram to
> see how everything works. Beware: it's very complicated.
>
> I can tell you that Tomcat isn't connecting to itself (unless your
> application is doing that). Unless some other process is connecting to
> your Tomcat's port 8080, it seems like nginx is the only possibility.
>
> Note: you can run lsof without a process parameter. You can search all
> open files for who owns file handle X and see who owns it. My guess is
> you'll get "kernel" if you look.
>
> - -chris
>
> > On Wed, 24 Jun 2020, 20:32 Ayub Khan, <ayub...@gmail.com> wrote:
> >
> >> Chris,
> >>
> >> Ok, I will investigate nginx side as well. Thank you for the
> >> pointers
> >>
> >> On Wed, 24 Jun 2020, 19:45 Christopher Schultz, <
> >> ch...@christopherschultz.net> wrote:
> >>
> >
> > Ayub,
> >
> > On 6/24/20 11:05, Ayub Khan wrote:
> >>>>> If some open file is owned by nginx why would it show up if
> >>>>> I run the below command> sudo lsof -p $(cat
> >>>>> /var/run/tomcat8.pid)
> >
> > Because network connections have two ends.
> >
> > -chris
> >
> >>>>> On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
> >>>>> ch...@christopherschultz.net> wrote:
> >>>>>
> >>>>> Ayub,
> >>>>>
> >>>>> On 6/23/20 19:17, Ayub Khan wrote:
> >>>>>>>> Yes we have nginx as reverse proxy, below is the
> >>>>>>>> nginx config. We notice this issue only when there is
> >>>>>>>> high number of requests, during non peak hours we do
> >>>>>>>> not see this issue.> location /myapp/myservice{
> >>>>>>>> #local machine proxy_pass http://localhost:8080;
> >>>>>>>> proxy_http_version 1.1;
> >>>>>>>>
> >>>>>>>> proxy_set_header Connection $connection_upgrade;
> >>>>>>>> proxy_set_header Upgrade $http_upgrade;
> >>>>>>>> proxy_set_header Host $host; proxy_set_header
> >>>>>>>> X-Real-IP $remote_addr; proxy_set_header
> >>>>>>>> X-Forwarded-For $proxy_add_x_forwarded_for;
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >>>>>
> >>>>> You might want to read about tuning nginx to drop
> >>>>> connections after a certain period of time, number of
> >>>>> requests, etc. Looks like either a bug in nginx or a
> >>>>> misconfiguration which allows connections to stick-around
> >>>>> like this. You may have to ask the nginx people. I have no
> >>>>> experience with nginx myself, while others here may have
> >>>>> some experience.
> >>>>>
> >>>>>>>> location / { # if using AWS Load balancer, this bit
> >>>>>>>> checks for the presence of the https proto flag. if
> >>>>>>>> regular http is found, then issue a redirect
> >>>>> to hit
> >>>>>>>> the https endpoint instead if
> >>>>>>>> ($http_x_forwarded_proto != 'https') { rewrite ^
> >>>>>>>> https://$host$request_uri? permanent; }
> >>>>>>>>
> >>>>>>>> proxy_pass http://127.0.0.1:8080;
> >>>>>>>> proxy_http_version 1.1;
> >>>>>>>>
> >>>>>>>> proxy_set_header Connection $connection_upgrade;
> >>>>>>>> proxy_set_header Upgrade $http_upgrade;
> >>>>>>>> proxy_set_header Host $host; proxy_set_header
> >>>>>>>> X-Real-IP $remote_addr; proxy_set_header
> >>>>>>>> X-Forwarded-For $proxy_add_x_forwarded_for;
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >>>>>>>>
> >>>>>>>> *below is the connector*
> >>>>>>>>
> >>>>>>>> <Connector port="8080"
> >>>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
> >>>>>>>> connectionTimeout="2000" maxThreads="50000"
> >>>>>>>> URIEncoding="UTF-8" redirectPort="8443" />
> >>>>>
> >>>>> 50k threads is a LOT of threads. Do you expect to handle
> >>>>> 50k requests simultaneously?
> >>>>>
> >>>>>>>> these ports are random, I am not sure who owns the
> >>>>>>>> process.
> >>>>>>>>
> >>>>>>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) ,
> >>>>>>>> here port 55866 is a random port.
> >>>>> I'm sure you'll find that 55866 is owned by nginx. netstat
> >>>>> will tell you .
> >>>>>
> >>>>> I think you need to look at your nginx configuration. It
> >>>>> would also be a great time to upgrade to a supported
> >>>>> version of Tomcat. I would recommend 8.5.56 or 9.0.36.
> >>>>>
> >>>>> -chris
> >>>>>
> >>>>>>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz
> >>>>>>>> < ch...@christopherschultz.net> wrote:
> >>>>>>>>
> >>>>>>>> Ayub,
> >>>>>>>>
> >>>>>>>> On 6/23/20 16:23, Ayub Khan wrote:
> >>>>>>>>>>> I executed *sudo lsof -p $(cat
> >>>>>>>>>>> /var/run/tomcat8.pid) *and I saw the below
> >>>>>>>>>>> output, some in CLOSE_WAIT and others in
> >>>>>>>>>>> ESTABLISHED. If there are 200 open file
> >>>>>>>>>>> descriptors 160 are in CLOSE_WAIT state. When
> >>>>>>>>>>> the count for CLOSE_WAIT increases I just have
> >>>>>>>>>>> to restart tomcat.
> >>>>>>>>>>>
> >>>>>>>>>>> java 65189 tomcat8 715u IPv6 237878311
> >>>>>>>>>>> 0t0 TCP localhost:http-alt->localhost:43760
> >>>>>>>>>>> (CLOSE_WAIT) java 65189 tomcat8 716u IPv6
> >>>>>>>>>>> 237848923 0t0 TCP
> >>>>>>>>>>> localhost:http-alt->localhost:40568
> >>>>>>>>>>> (CLOSE_WAIT)
> >>>>>>>>
> >>>>>>>> These are connections from some process into Tomcat
> >>>>>>>> listening on port 8080 (that's what
> >>>>>>>> localhost:http-alt is). So what process owns the
> >>>>>>>> outgoing connection on port 40568 on the same host?
> >>>>>>>>
> >>>>>>>> Are you using a reverse proxy?
> >>>>>>>>
> >>>>>>>>>>> most of the open files are in CLOSE_WAIT state
> >>>>>>>>>>> I do not see anything related to database ip.
> >>>>>>>>
> >>>>>>>> Agreed. It looks like you have a reverse proxy who is
> >>>>>>>> losing-track of connections, or who is (re)opening
> >>>>>>>> connections when it may be unnecessar y.
> >>>>>>>>
> >>>>>>>> Can you share your <Connector> configuration from
> >>>>>>>> server.xml? Remember to remove any secrets.
> >>>>>>>>
> >>>>>>>> -chris
> >>>>>>>>
> >>>>>>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix
> >>>>>>>>>>> Schumacher < felix.schumac...@internetallee.de>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> >>>>>>>>>>>>> Felix,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I executed ls -l /proc/$(cat
> >>>>>>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the
> >>>>>>>>>>>> output
> >>>>>>>>>>>>> I see majority of them are related to
> >>>>>>>>>>>>> sockets as shown below, some of
> >>>>>>>>>>>> them
> >>>>>>>>>>>>> point to the jar file of tomcat and others
> >>>>>>>>>>>>> to the log file which is
> >>>>>>>>>>>> created.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> socket:[2084570754] socket:[2084579487]
> >>>>>>>>>>>>> socket:[2084578478] socket:[2084570167]
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you try the other command (lsof -p $(cat
> >>>>>>>>>>>> ...tomcat.pid))? It should give a bit more
> >>>>>>>>>>>> details on the used sockets that the proc
> >>>>>>>>>>>> directory.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Felix
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix
> >>>>>>>>>>>>> Schumacher <
> >>>>>>>>>>>>> felix.schumac...@internetallee.de> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>>>>>>>>>>>>>> Chris,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am using HikariCP for connection
> >>>>>>>>>>>>>>> pooling. If the database is leaking
> >>>>>>>>>>>>>>> connections then I should see
> >>>>>>>>>>>>>>> connection not available exception.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> How do I find out which file
> >>>>>>>>>>>>>>> descriptors are leaking ? these are
> >>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>> files
> >>>>>>>>>>>>>>> open on disk as there is no explicit
> >>>>>>>>>>>>>>> disk file I/O in this application.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I just use the below command to check
> >>>>>>>>>>>>>>> for open file descriptors:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
> >>>>>>>>>>>>>> You could have a look at the name of the
> >>>>>>>>>>>>>> files in the pids proc
> >>>>>>>>>>>> directory.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> $ ls -l /proc/$(cat
> >>>>>>>>>>>>>> /var/run/tomcat8.pid)/fd/
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Or you could use the tool lsof to find
> >>>>>>>>>>>>>> the open file descriptors.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> For both calls you should first change
> >>>>>>>>>>>>>> to the uid of the tomcat user or use sudo
> >>>>>>>>>>>>>> as in your example.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Felix
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks and Regards Ayub
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM
> >>>>>>>>>>>>>>> Christopher Schultz <
> >>>>>>>>>>>>>>> ch...@christopherschultz.net> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Ayub,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>>>>>>>>>>>>>> Sorry we are using 8.0.32
> >>>>>>>>>>>>>>>>>> version of tomcat.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> below is the configuration:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Server version: Apache
> >>>>>>>>>>>>>>>>>> Tomcat/8.0.32 (Ubuntu) Server
> >>>>>>>>>>>>>>>>>> built: Jan 24 2020 16:24:30
> >>>>>>>>>>>>>>>>>> UTC Server number: 8.0.32.0 OS
> >>>>>>>>>>>>>>>>>> Name: Linux OS Version:
> >>>>>>>>>>>>>>>>>> 4.4.0-1087-aws Architecture:
> >>>>>>>>>>>>>>>>>> amd64 JVM Version: 1.8.0_181-b13
> >>>>>>>>>>>>>>>>>> JVM Vendor: Oracle Corporation
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I use the below command to check
> >>>>>>>>>>>>>>>>>> the file descriptors:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc
> >>>>>>>>>>>>>>>>>> -l"
> >>>>>>>>>>>>>>> So you know there is some kind of
> >>>>>>>>>>>>>>> increase in file-handle use, but you
> >>>>>>>>>>>>>>> don't know what types of file handles
> >>>>>>>>>>>>>>> are increasing, right?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Can you try to find out which kinds of
> >>>>>>>>>>>>>>> file handles are increasing?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have a sneaking suspicion that it's
> >>>>>>>>>>>>>>> your database connections and not
> >>>>>>>>>>>>>>> actually files open on the disk.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Are you using a database connection
> >>>>>>>>>>>>>>> pool? If not, you should really use one
> >>>>>>>>>>>>>>> and limit the number of connections to
> >>>>>>>>>>>>>>> something sane. If you are using one,
> >>>>>>>>>>>>>>> are you monitoring it to see how many
> >>>>>>>>>>>>>>> connections are actually being used?
> >>>>>>>>>>>>>>> Are you sure you are using proper
> >>>>>>>>>>>>>>> resource management[1]? Even a single
> >>>>>>>>>>>>>>> code-path that leaks connections can
> >>>>>>>>>>>>>>> leak them quickly under load.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> When there an issue related to
> >>>>>>>>>>>>>>>>>> broken files, this value keeps
> >>>>>>>>>>>>>>>>>> increasing, the only way to
> >>>>>>>>>>>>>>>>>> bring it down is to remove vm
> >>>>>>>>>>>>>>>>>> instance from AWS load balancer.>
> >>>>>>>>>>>>>>>>>> Which version of tomcat should I
> >>>>>>>>>>>>>>>>>> install ?
> >>>>>>>>>>>>>>> Tomcat 8.0.x hasn't been supported
> >>>>>>>>>>>>>>> since its last release on 29 June 2018.
> >>>>>>>>>>>>>>> That was 8.0.53. Your release is from
> >>>>>>>>>>>>>>> 8 February 2016 and is dangerously out
> >>>>>>>>>>>>>>> of date (unless you are using the
> >>>>>>>>>>>>>>> Ubuntu-packaged version, in which case
> >>>>>>>>>>>>>>> I hope they kept-up with security
> >>>>>>>>>>>>>>> patches thee past 4 years).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM
> >>>>>>>>>>>>>>>>>> Christopher Schultz <
> >>>>>>>>>>>>>>>>>> ch...@christopherschultz.net>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Ayub,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe
> >>>>>>>>>>>>>>>>>>>>> increases open files on
> >>>>>>>>>>>>>>>>>>>>> ubuntu AWS
> >>>>>>>>>>>>>>>>>> Which exact version of Tomcat
> >>>>>>>>>>>>>>>>>> 8.5? If you aren't running the
> >>>>>>>>>>>>>>>>>> latest version (8.5.56), please
> >>>>>>>>>>>>>>>>>> upgrade and re-test.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> If there is slow response
> >>>>>>>>>>>>>>>>>>>>> from db I see this stack
> >>>>>>>>>>>>>>>>>>>>> trace and the open files
> >>>>>>>>>>>>>>>>>>>>> goes high and the only way
> >>>>>>>>>>>>>>>>>>>>> to open files go down is to
> >>>>>>>>>>>>>>>>>>>>> remove the instance from
> >>>>>>>>>>>>>>>>>>>>> Amazon load balancer.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Is there a way to keep the
> >>>>>>>>>>>>>>>>>>>>> open files low even when
> >>>>>>>>>>>>>>>>>>>>> Broken pipe error is
> >>>>>>>>>>>>>>>>>>>>> thrown ?
> >>>>>>>>>>>>>>>>>> What is your evidence that file
> >>>>>>>>>>>>>>>>>> handles are being left open?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Which file handles are being
> >>>>>>>>>>>>>>>>>> left open?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -chris
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---------------------------------------------------------
> - ----
> >
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> - ---
> >>>>>
> >>>>>>>>>>>>>>
> > ---
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>> ------------------------------------------------------------------
> - ---
> >>>>>>>>>
> >>>>>
> >
> >>>>>
> >>>>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For
> >>>>>>>>> additional commands, e-mail:
> >>>>>>>>> users-h...@tomcat.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>> -----------------------------------------------------------------
> - ----
> >>>>>>
> >>>>>>
> >
> >>>>>>
> >>>>>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>>>> For additional commands, e-mail:
> >>>>>> users-h...@tomcat.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>> --------------------------------------------------------------------
> - -
> >>>
> >>>
> >>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl703BAACgkQHPApP6U8
> pFgxGA/7BndNw95OCQCgrR6Oi6QQc6z5XOPkLQfD11r442fpFI/pjJQVI0830Nzm
> JNOVmBe1zPYTxn1k3yl7YMmjQjNWynMozNjgOcphcgEybGTJD9AndCPrcO/qeM7j
> Retpr4SYoa/GcQ3Gd17U52QeM5hYgvGQFYso36UWT10YLvlixaq7ftEagbArvEgX
> b33Dmr78mcI+NvMnkp267S8iLT+3kzbI8yYjhGxhHGOVXIENocjgFhHdLM4giKk3
> XZm8jQnpe2qRaVss1bsMsSSI1ip6x1miwwr9SuEIwUMeHkp0PpwpIu9ETPt+4iSD
> ip232ftdk42n07uLNWCPf7J/oQHtOweBpTgQ2c6lGnUlJt+hAu8dl09z+NaROFDu
> 78Pbmaq5PX3pAwFEexVGAzk/u/tnz073kSlMKEK/CIMFqeVRb/96vg+Rqxqus/u5
> SCPu94GxtkrcMRSR9SXDaunFqEMrNRdQHQUvHHLltq5W5esk8HM2d+yNyZ+EyYuX
> fzVDDYcl9qbbolokslkB25uw1Hl7zeVgdHRpJkcxHBt3TYLv+QkDUZbDUb2ZWwSC
> +oTexRE0H4MWZfGw9Dg8UjH0y2th9z9HJWNGSBICEO9RghFqLttopFD+ej9RiUEo
> P47a4wz7MAR5cZbVQnKakEPUb4N+SGXW5ZHHvZAw4oDnxK1fF4Q=
> =F5XJ
> -----END PGP SIGNATURE-----
>
--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
