Re: Crash in http connector random once a day
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Daniel and Rainer,
On 3/15/19 07:56, Daniel Castilla | thin(k)design wrote:
>> In each of the four thread dumps, there are (the same) 4 threads
>> connected to and waiting for an answer from a PHP FCGI backend
>> behind Tomcat. You need to investigate, why that backend isn't
>> sending a response (or takes so long for it).
>>
>> Regards,
>>
>> Rainer
>>
>
> Hi Rainer,
>
> I have debugged PHP, the process ends normally and the response is
> sent 100% of the time, so I don't know why the tomcat thread keeps
> listening to it.
>
> There are always a maximum of 4 threads that keeps reading from
> the PHP Input Stream. As long as there are 4 "hanging" threads all
> other requests to tomcat work perfectly, which is very strange.
>
> The thread dumps always show this:
>
> java.lang.Thread.State: RUNNABLE at
> java.net.SocketInputStream.socketRead0(Native Method) at
> java.net.SocketInputStream.read(Unknown Source) at
> java.net.SocketInputStream.read(Unknown Source) at
> php.java.fastcgi.FCGIInputStream.read(FCGIInputStream.java:39) ...
>
> At some point (up to 24 hours) tomcat "cleans" the threads and
> they stop. And after a while it begins with the failures until
> there are again 4 "hanging" threads.
>
> If I look in VisualVM in the tab MBeans at the PhpCGIServlet of
> those processes, there is a maxTime of 3406, so PHP should not be
> the problem.
It's unclear to me whether the at
php.java.servlet.fastcgi.FastCGIServlet.doPost(FastCGIServlet.java:491)
method
is *handling* a POST -- that is, responding to a POST request to
Tomcat or *initiating* a POST to another service. I suspect the
answer to the above question will help understand what is going on, here
.
I find it strange that there is code doing "PHP FastCGI" running
within a Java servlet container... what's wrong with regular HTTP?
- -chris
>> Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:
> From: Daniel Castilla | thin(k)design
> [mailto:d...@thin-k-design.com] Sent: Thursday, March 14,
> 2019 12:37 PM To: Jäkel, Guido ;
> users@tomcat.apache.org Subject: Re: Crash in http
> connector random once a day
>
> Dear Guido,
>
> thanks for the reply. The requests are reaching tomcat, and
> a thread is always started, if I look at the current
> threads on the tomcat manager I see the following, there
> are 4 threads that are processing since 2+ hours:
>
> R ? ? ? ? ? ? S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1
> 0:0:0:0:0:0:0:1 localhost GET /manager/status HTTP/1.1 S
> 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1
> localhost POST /cloudworx/?method=words=17385 HTTP/1.1 S
> 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1
> localhost POST /cloudworx/?method=words=18986 HTTP/1.1 S
> 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1
> localhost POST /cloudworx/?method=words=10560 HTTP/1.1 R
> ? ? ? ? ? ? S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1
> 0:0:0:0:0:0:0:1 localhost POST
> >/cloudworx/?method=words=10560 HTTP/1.1
>
> I'm not sure what other metrics would be helpful, but your
> Unix script wouldn't help much, as I am on a Windows Server
> 2012 and I would like to avoid installing Cygwin or
> something similar.
Dear Daniel,
the script just read-out the same core data and does some
pretty print. You may do it by your own by doing a
HTTP-Request against
URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Cat
alina:type=RequestProcessor,*'
What is interesting from your snippet that there are POST Request "in
Service" (i.e. Progress) since more than 2h. And you told that the ID is
unique, but there are two times a '10560'.
Is there a database service involved in the backend?
BTW: In addition to pull static thread dumps you may use
JVisualVM to get a live view to the threads.
>>>
>>> Dear Guido and Mark,
>>>
>>> the same Id '10560' appears twice in the URL as the same
>>> request was retried for two times, the third time it worked.
>>> The other two URLs had only one retry, the second time it
>>> worked.
>>>
>>> You can find a complete thread dump here (I hope the link gets
>>> through):
>>> https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/vi
ew?usp=sharing
>>>
>>>
>>>
I had to restart tomcat now in order to access the jmxproxy as it
>>> wasn't active in the user roles, so the active "problem"
>>> processes are gone for now. But it will happen again, at latest
>>> tomorrow.
>>>
>>> I tried JVisualVM, but I am no Java Expert and don't know what
>>> to search for...
>>>
>>> Thanks Daniel
>
> -
>
>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-BEGIN PGP SIGNATURE-
Re: [OT] Crash in http connector random once a day
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Daniel, On 3/14/19 06:09, Daniel Castilla | thin(k)design wrote: > Hi, > > I have long searched and haven't found any real hint how to solve > my problem, so I am adressing you. > > Here are my specifications: > > Server version:Apache Tomcat/8.5.15 Server built: > May 5 2017 11:03:04 UTC Server number: 8.5.15.0 OS Name: > Windows Server 2012 OS Version:6.2 Architecture: > amd64 Java Home: C:\Program Files\Java\jre7 JVM > Version: 1.7.0_25-b17 JVM Vendor:Oracle > Corporation > > The tomcat connector I am using has no custom configuration: > > > > I have a PHP Javabridge app (runs PHP CGI to access JAVA > functions) running on tomcat and it's beeing accessed solely > through localhost from other PHP scripts running on IIS. It goes as > follows: > > 1. Request sent to IIS 2. Run PHP on IIS, process the request and > delegates a part of the processing to tomcat 3. Access tomcat > through HTTP on localhost:8080 and wait maximum 60s for the > response, retry up to 3 times I'm curious, if you are willing to wait for 3 minutes for a response from the Tomcat server, why bother closing the request after 1 minute and re-trying? That could potentially turn a single long-running request into 3 simultaneous long-running requests, which may cause a cascade and take-down lots other requests. Why not simply wait up to 3 minutes a single time? > In the JAVA logs I can see a "java.net.SocketException: Connection > reset", which is logical, as the PHP process on IIS closed the > connection after the 60s timeout. I tried to set this timeout to > infinite, with the result that the PHP process on IIS waited > between 8 and 24 hours, only to get a 500 from tomcat. So Tomcat will wait up to 24 hours before returning the 500 response? That seems .. unusual. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyMQHAACgkQHPApP6U8 pFjRKBAAj2XN2+93wf3+eLxnBUvMSEoAmPEBqHRXAaIWRWxzUKLBZnq+BjjqccH6 Krlc9OemUIVAurwtOGuKx4erD+Iy2bCxAlit9NkeGJLKmPM1l90NXyglWNNqxZZD 1c5oXSm6Bmfws6Cw4Z0geEzRqHRWS6mBDl/W1XCmiK9wUMWmxFkol2xOrv74lMrO m97ZK8uEEBFzFLqT4lF/n2SbP2A1BJjBfqCC44jBvAqlYN1DuXY/yIkjaIY/+Haw ev0CJyjaICfu96QBNdQKvAnQTwHIUPy5SYBEQbgqyxXfiwbw7Gx+KXnSK0DK9N7b Zs4VI93/R8y2+BhFZ8JjQjh0cJyJKxSl2TkL/CFkOdaWEFmRlqBZ6LDkujENFW6s 39x1Jni553+ISPe84nffCheIkGQDin/kP11cgRyJgmNyFTpXBKKTZkEy8zssJy28 Q2dLSMe3OIgpqVPOXeq/zb+uWwf/XOA4nBPVjY9MLuuZAedirpmHgzZgdCjNK/IT eEZUjxFui56uvJTr4vQEf/dwMQVQB82KxezPaHD/qC3zldoVA86UWBWomwy/MeYb Eaf+iHeOBZ2h4q+TJY3fy2wjd8xMHlLpPmtPY1+uBe6HzvuID/ytg9uoGMbWkXoe w4Iaif3wPHItJld935Ccifi1UPhT/0N31GC6hVR+U7wbR9QlP2U= =b7Gb -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 instance of ParallelWebAppClassLoaders in Tomcat JVM
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 3/15/19 19:49, Mark Thomas wrote: > On 15/03/2019 17:51, Shankar P S wrote: >> @Mark Thanks for the tip. I compared the 2 heap dumps. For the >> same server.xml config, 8.5.38 version of Tomcat creates 3 >> StandardContext instances. I see that these 3 objects have their >> own hierarchy. *Tomcat 8.5.38 - 3 instances of >> StandardContext* >> >> First instance originalDocBase >> "/var/www/tomcat/webapps/ROOT.war", docBase "ROOT", displayName >> "/", workDir "work/Catalina/localhost/ROOT" >> >> Second instance docBase ROOT.war, workDir >> "work/Catalina/localhost/ROOT" >> >> Third instance originalDocBase >> "/var/www/tomcat/webapps/ROOT.war", docBase "ROOT", displayName >> "/", workDir "work/Catalina/localhost/ROOT" > > Odd. Worth checking the state field for each of those instances. > > Check the logs. You should see start (and hopefully some stop) > messages for those. > > Is it possible the app has been reloaded? +1 Shankar, if it's possible to modify the web application, you might want to use a ServletContextListener to purge that cache on application shutdown. If the application itself has a ClassLoader leak (which is common), cleaning-up a little bit on shutdown can stop the bleeding while you investigate the root cause of any ClassLoader leak. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyMPLQACgkQHPApP6U8 pFiWxQ//aAopKwEpHy95eoGPIFZfTW28Wif3dWyk+oE2RDy+0hQaOROgHDvqUVCC VOSbApqAwV/jvE+VW1fFrML8T3hVTbxN2oRPYjjljbMyYAaQMiu2sFX8u0+I2O+A XT+BuIA1Sf0ZzeNEd7X892rdG5omH7TL8w4Xa1errsXxwUaTVr2l6QqZfpLpRSRP uBjUXdH/Qkxu0eJ55pPxgfmgdfOU28fE7bd+absloXYitCxW6KnyxgL/GCaFq44t eSmOfxNI/7RQc64x/R+niPF3i1CDFZreEuBu6+2negPwWgQNGQrSTyQY2jnB8ZOa 5weROOdOQtKlr1Nw6wW/y3Asx9ti/lDBpbzyDLMRe+q4PGV/A2t2mRoNDdmPdHje famrH3kk0B6ijn4rVe2HO68ca/sJuMLMP8itrj72+gAWEEGorzkSBBJg7Hr2Czzm dIjfCqV8u1dVCmexn+dYc990gHp/TvnUCEZpFuySdq1le8yKLU2BhRBiZA88uR0m F8rRYrAQZDHZ6GOcQLS8C261O58HBPglOuatgej5EW0Ys7MqLnOh6IeK5NKpjLdT dyGkOBCB7sqC/1DYmfT5EJlhzTJ/tbn/8kyfZfJOKBGgr+xZv23tfVHVNv1LJuGJ J/2k6IRcQzariMTCTUTmU7I/oXfsoK8lffSGEmfjMGYV+slDQs8= =66tB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat doesn't restart
On 14/03/2019 20:23, Louis Zipes wrote: > Hi Experts, > Running Tomcat 7.0.54 on Windows 2012 > > Our morning process is to fire a stop of the Tomcat Windows Service and then > an immediate restart. This is more due to the 3rd party application that is > running with Tomcat than a problem with Tomcat. A few weeks ago it started > to not cleanly stop when using the following batch file command. Note that > this had worked for more than a year with no problems and I don't think that > there have been any server changes although I don't control the machine > > > net stop "Apache Tomcat 7.0 TPMDEV" && net start "Apache Tomcat 7.0 TPMDEV" > > I then changed the batch file, that calls the Windows Service stop/start to > give a 65 second delay between the stop and start hoping to give it more time > to cleanly stop before restarting: > > > > > net stop "Apache Tomcat 7.0 TPMDEV" > > PING localhost -n 65 >NUL > > net start "Apache Tomcat 7.0 TPMDEV" > > > > > This modified batch file worked for about 2 weeks but then this morning it > failed to stop and restart correctly. Since it is PRD we just have to get it > restarted so I usually don't have time to troubleshoot getting threads, etc. > (also, the off shore team is on shift when this happens but I'm the > administrator that actually has to get to root cause when I start my shift). > > I observed the following in the std-output log it seems to receive the > Shutdown command but it seems like either all of the threads are not being > closed for some reason > > Mar 14, 2019 5:28:00 AM org.apache.catalina.core.StandardServer await > INFO: A valid shutdown command was received via the shutdown port. Stopping > the Server instance. > Mar 14, 2019 5:28:00 AM org.apache.coyote.AbstractProtocol pause > INFO: Pausing ProtocolHandler ["http-bio-7005"] > Mar 14, 2019 5:28:00 AM org.apache.catalina.core.StandardService stopInternal > INFO: Stopping service Catalina > Mar 14, 2019 5:28:00 AM org.apache.catalina.core.StandardWrapper unload > INFO: Waiting for 94 instance(s) to be deallocated for Servlet > [RequestDispatcherServlet] > Mar 14, 2019 5:28:01 AM org.apache.catalina.core.StandardWrapper unload > INFO: Waiting for 94 instance(s) to be deallocated for Servlet > [RequestDispatcherServlet] > Mar 14, 2019 5:28:03 AM org.apache.catalina.core.StandardWrapper unload > INFO: Waiting for 94 instance(s) to be deallocated for Servlet > [RequestDispatcherServlet] > -- > > > So my questions are: > > 1) Does anyone have a better way to make sure Tomcat is fully stopped and any > open threads are killed Killing the process is probably the only other option. > 2) Is this part below real cause and I need to find the root cause by doing a > thread dump? As mentioned it is hard for me to get one because I'm usually > not on shift when this happens. Yes. You need a thread dump to find out what is going on here. I'd guess the application has been updated just before this problem started happening. Mark > > Mar 14, 2019 5:28:03 AM org.apache.catalina.core.StandardWrapper unload > INFO: Waiting for 94 instance(s) to be deallocated for Servlet > [RequestDispatcherServlet] > > Thanks, Louis > > --- > CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and > may contain information that is confidential, proprietary or exempt from > disclosure. If you are not the intended recipient, please contact the sender > immediately. Unauthorized use or distribution is prohibited and may be > unlawful. > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 instance of ParallelWebAppClassLoaders in Tomcat JVM
On 15/03/2019 17:51, Shankar P S wrote: > @Mark Thanks for the tip. I compared the 2 heap dumps. For the same > server.xml config, 8.5.38 version of Tomcat creates 3 StandardContext > instances. I see that these 3 objects have their own hierarchy. > > *Tomcat 8.5.38 - 3 instances of StandardContext* > > First instance originalDocBase "/var/www/tomcat/webapps/ROOT.war", > docBase "ROOT", displayName "/", workDir "work/Catalina/localhost/ROOT" > > Second instance docBase ROOT.war, workDir "work/Catalina/localhost/ROOT" > > Third instance originalDocBase "/var/www/tomcat/webapps/ROOT.war", > docBase "ROOT", displayName "/", workDir "work/Catalina/localhost/ROOT" Odd. Worth checking the state field for each of those instances. Check the logs. You should see start (and hopefully some stop) messages for those. Is it possible the app has been reloaded? Mark > > > *Tomcat 8.5.35 - 1 instance of StandardContext* > > First instance originalDocBase "/var/www/tomcat/webapps/ROOT.war", > docBase "ROOT", displayName "/", workDir "work/Catalina/localhost/ROOT" > - > The server.xml & context.xml is the same and they have very few custom > configs. How can I debug from here? > > > > On Fri, Mar 15, 2019 at 1:30 AM Mark Thomas wrote: > >> On 14/03/2019 21:15, Shankar P S wrote: >>> Version - Tomcat 8.5.38 >>> OS - Centos 7.6 >>> JVM - openjdk 1.8, 4 CPU >>> >>> We recently upgraded an existing server running a Spring 3.0 web app from >>> Tomcat 8.5.35 to 8.5.38. >>> >>> After the upgrade, I noticed OutOfMemory exceptions several times. >>> Increasing the heap size (-Xmx) several times upto nearly twice (from 6 >> GB >>> to 10 GB) helped bring the problem under control. >>> >>> I was worried about a memory leak and so took a heap dump. Upon >> inspection >>> I noticed that there were 2 copies of 2 large HashMap objects that we use >>> as a cache for fast lookup. I noticed that this was because there were 2 >>> instances of org.apache.catalina.loader.ParallelWebappClassLoader. >>> >>> Downgrading to 5.3.35, reduced the heap size to half and the problem was >>> gone. In the new heap dump, there was only one instance of >>> ParallelWebappClassLoader. >>> >>> My question is - why does Tomcat 5.3.38 create 2 instances of >>> ParallelWebappClassLoader, while 5.3.35 creates only one?Is there a way >> to >>> conigure the number of instances of ParallelWebappClassLoaderwhen >>> upgrading to 8.5.38? >> >> That sounds like you have double deployment. That normally means >> configuration error. >> >> Since you have a heap dump, look for StandardContext instances and then >> look at the value of docBase for each. That should give you some >> pointers to what is happening. >> >> Mark >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 instance of ParallelWebAppClassLoaders in Tomcat JVM
@Mark Thanks for the tip. I compared the 2 heap dumps. For the same server.xml config, 8.5.38 version of Tomcat creates 3 StandardContext instances. I see that these 3 objects have their own hierarchy. *Tomcat 8.5.38 - 3 instances of StandardContext* First instance originalDocBase "/var/www/tomcat/webapps/ROOT.war", docBase "ROOT", displayName "/", workDir "work/Catalina/localhost/ROOT" Second instance docBase ROOT.war, workDir "work/Catalina/localhost/ROOT" Third instance originalDocBase "/var/www/tomcat/webapps/ROOT.war", docBase "ROOT", displayName "/", workDir "work/Catalina/localhost/ROOT" *Tomcat 8.5.35 - 1 instance of StandardContext* First instance originalDocBase "/var/www/tomcat/webapps/ROOT.war", docBase "ROOT", displayName "/", workDir "work/Catalina/localhost/ROOT" - The server.xml & context.xml is the same and they have very few custom configs. How can I debug from here? On Fri, Mar 15, 2019 at 1:30 AM Mark Thomas wrote: > On 14/03/2019 21:15, Shankar P S wrote: > > Version - Tomcat 8.5.38 > > OS - Centos 7.6 > > JVM - openjdk 1.8, 4 CPU > > > > We recently upgraded an existing server running a Spring 3.0 web app from > > Tomcat 8.5.35 to 8.5.38. > > > > After the upgrade, I noticed OutOfMemory exceptions several times. > > Increasing the heap size (-Xmx) several times upto nearly twice (from 6 > GB > > to 10 GB) helped bring the problem under control. > > > > I was worried about a memory leak and so took a heap dump. Upon > inspection > > I noticed that there were 2 copies of 2 large HashMap objects that we use > > as a cache for fast lookup. I noticed that this was because there were 2 > > instances of org.apache.catalina.loader.ParallelWebappClassLoader. > > > > Downgrading to 5.3.35, reduced the heap size to half and the problem was > > gone. In the new heap dump, there was only one instance of > > ParallelWebappClassLoader. > > > > My question is - why does Tomcat 5.3.38 create 2 instances of > > ParallelWebappClassLoader, while 5.3.35 creates only one?Is there a way > to > > conigure the number of instances of ParallelWebappClassLoaderwhen > > upgrading to 8.5.38? > > That sounds like you have double deployment. That normally means > configuration error. > > Since you have a heap dump, look for StandardContext instances and then > look at the value of docBase for each. That should give you some > pointers to what is happening. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: TLS protocols and cipher suites available under JSSE?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 James, On 3/13/19 20:38, James H. H. Lampert wrote: > Thanks, Mr. Schultz. > > I managed to find the IBM docs. At least some of the cipher suites > the customer is talking about are supported all the way back to > their 7.0 JVM. > > I've specified cipher suites by name in the connector, but I don't > think I've done protocols. "TLS," according to the IBM docs, > "Enables TLS v1.0 protocol (defined in RFC 2246). Accepts TLSv1 > hello encapsulated in an SSLv2 format hello." > > Sounds like I would need to enable TLSv1.1 and TLSv1.2 explicitly. > Would I change the sslProtocol clause from >> sslProtocol="TLS" > > to >> sslProtocol="TLSv1.1,TLSv1.2" > or even >> sslProtocol="TLSv1.2" > > and specify acceptable cipher suites in the ciphers clause? Specifying sslProtocol="TLS" is what you want, regardless of the actual protocols you want to support. It's ... odd, but you have to tell Java that you want "TLS" (generically) before you can tell it what /kind/ of TLS you want. I believe that Java will never accept any value other than "TLS" for that at this point. So you are left with sslEnabledProtocols... If you want to configure specific versions of TLS, this is where to do i t: sslEnabledProtocols="TLSv1.2, TLSv1.1, SSLv2Hello" (etc) If you can tolerate it, I'd enable only TLSv1.2. Some clients still require TLSv1. Virtually all clients supporting TLSv1.1 also support TLSv1.2, so it's almost useless to enable TLSv1.1. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyLsFgACgkQHPApP6U8 pFirkQ//TA0sE1AVpW7dr1/1+zkKZ/n9AIm6nKEnamcN6fF0rDchP+CW/uZMJjrF TuKQUEyOZjS8M7qU7t6HR7xdwt7JbpPQhZ07PYLqhxw0uNYfZnqQfc1AWz+pMLxD RNWPrtb9+A4uWI5B/c64rVJjD3Ig/dAZHH5xesspZrEwt12bTBVKdSa610pnN7Dr DFY3PSpxeF+86a5/SJwZFEU1iu36+8krA7H99qA4XBYiwC4UPxIoSqtNznmZ08dH kRrJ3EBSbDFB7/h/CqJUCnUyfJscal5ZYsALp/DKt1xpK7R3nGduRbgpsud5NcbD bl9RNoEZ2aQBJClO+FLumWPnOX2pqLqIB9QxSdqxdC9kRUiSflcAc1XJXtWcHWty PRXHEneG6AMe4uqJ7nzlxOBkCQlF7G3aShkbqPmmzV/IV0H3ncVt/4qW4fS+N5AR iv41tva/9M8Y39uFyWyvp0U5s6sby3xXC9jlI3aqdIkofxvTG1G7cNGZeGMHDxsX JY2e5k8/ZxKgaGifp6GWPNFt5fSiXpvVpJzQcBojwP4TP8D2cnxIyCZjPg0tUXiD U4gnJqA0b5oexPFFCA1qZUTl0xgBCeZxqS8NcDh8Nu+hb4N0ZT8tL/UvApD8ZmfB 2E7A3K+ny462SI6mcvTHMKIR1SPSYTQ3MD+yYvqtw3XOBCIR9Zg= =/zZH -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Crash in http connector random once a day
>
> In each of the four thread dumps, there are (the same) 4 threads
> connected to and waiting for an answer from a PHP FCGI backend behind
> Tomcat. You need to investigate, why that backend isn't sending a
> response (or takes so long for it).
>
> Regards,
>
> Rainer
>
Hi Rainer,
I have debugged PHP, the process ends normally and the response is
sent 100% of the time, so I don't know why the tomcat thread keeps
listening to it.
There are always a maximum of 4 threads that keeps reading from the
PHP Input Stream. As long as there are 4 "hanging" threads all other
requests to tomcat work perfectly, which is very strange.
The thread dumps always show this:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at php.java.fastcgi.FCGIInputStream.read(FCGIInputStream.java:39)
...
At some point (up to 24 hours) tomcat "cleans" the threads and they
stop. And after a while it begins with the failures until there are
again 4 "hanging" threads.
If I look in VisualVM in the tab MBeans at the PhpCGIServlet of those
processes, there is a maxTime of 3406, so PHP should not be the
problem.
A heapdump shows nothing unusual I think:
Total Bytes: 35.381.250
Total Classes: 6.655
Total Instances: 454.066
Classloaders: 178
GC Roots: 3.318
Number of Objects Pending for Finalization: 0
I am really out of ideas...
> Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:
> >>> From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >>> Sent: Thursday, March 14, 2019 12:37 PM
> >>> To: Jäkel, Guido ; users@tomcat.apache.org
> >>> Subject: Re: Crash in http connector random once a day
> >>>
> >>> Dear Guido,
> >>>
> >>> thanks for the reply. The requests are reaching tomcat, and a thread
> >>> is always started, if I look at the current threads on the tomcat
> >>> manager I see the following, there are 4 threads that are processing
> >>> since 2+ hours:
> >>>
> >>> R ? ? ? ? ? ?
> >>> S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET
> >>> /manager/status HTTP/1.1
> >>> S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> /cloudworx/?method=words=17385 HTTP/1.1
> >>> S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> /cloudworx/?method=words=18986 HTTP/1.1
> >>> S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> /cloudworx/?method=words=10560 HTTP/1.1
> >>> R ? ? ? ? ? ?
> >>> S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> >/cloudworx/?method=words=10560 HTTP/1.1
> >>>
> >>> I'm not sure what other metrics would be helpful, but your Unix script
> >>> wouldn't help much, as I am on a Windows Server 2012 and I would like
> >>> to avoid installing Cygwin or something similar.
> >>
> >> Dear Daniel,
> >>
> >> the script just read-out the same core data and does some pretty print.
> >> You may do it by your own by doing a HTTP-Request against
> >>
> >>
> >> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
> >>
> >>
> >> What is interesting from your snippet that there are POST Request "in
> >> Service" (i.e. Progress) since more than 2h. And you told that the ID is
> >> unique, but there are two times a '10560'.
> >>
> >> Is there a database service involved in the backend?
> >>
> >> BTW: In addition to pull static thread dumps you may use JVisualVM to get
> >> a live view to the threads.
> >
> > Dear Guido and Mark,
> >
> > the same Id '10560' appears twice in the URL as the same request was
> > retried for two times, the third time it worked.
> > The other two URLs had only one retry, the second time it worked.
> >
> > You can find a complete thread dump here (I hope the link gets
> > through):
> > https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing
> >
> > I had to restart tomcat now in order to access the jmxproxy as it
> > wasn't active in the user roles, so the active "problem" processes are
> > gone for now. But it will happen again, at latest tomorrow.
> >
> > I tried JVisualVM, but I am no Java Expert and don't know what to search
> > for...
> >
> > Thanks
> > Daniel
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 2 instance of ParallelWebAppClassLoaders in Tomcat JVM
On 14/03/2019 21:15, Shankar P S wrote: > Version - Tomcat 8.5.38 > OS - Centos 7.6 > JVM - openjdk 1.8, 4 CPU > > We recently upgraded an existing server running a Spring 3.0 web app from > Tomcat 8.5.35 to 8.5.38. > > After the upgrade, I noticed OutOfMemory exceptions several times. > Increasing the heap size (-Xmx) several times upto nearly twice (from 6 GB > to 10 GB) helped bring the problem under control. > > I was worried about a memory leak and so took a heap dump. Upon inspection > I noticed that there were 2 copies of 2 large HashMap objects that we use > as a cache for fast lookup. I noticed that this was because there were 2 > instances of org.apache.catalina.loader.ParallelWebappClassLoader. > > Downgrading to 5.3.35, reduced the heap size to half and the problem was > gone. In the new heap dump, there was only one instance of > ParallelWebappClassLoader. > > My question is - why does Tomcat 5.3.38 create 2 instances of > ParallelWebappClassLoader, while 5.3.35 creates only one?Is there a way to > conigure the number of instances of ParallelWebappClassLoaderwhen > upgrading to 8.5.38? That sounds like you have double deployment. That normally means configuration error. Since you have a heap dump, look for StandardContext instances and then look at the value of docBase for each. That should give you some pointers to what is happening. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
