Re: Crash in http connector random once a day

2019-03-15 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Daniel and Rainer,

On 3/15/19 07:56, Daniel Castilla | thin(k)design wrote:
>> In each of the four thread dumps, there are (the same) 4 threads 
>> connected to and waiting for an answer from a PHP FCGI backend
>> behind Tomcat. You need to investigate, why that backend isn't
>> sending a response (or takes so long for it).
>> 
>> Regards,
>> 
>> Rainer
>> 
> 
> Hi Rainer,
> 
> I have debugged PHP, the process ends normally and the response is 
> sent 100% of the time, so I don't know why the tomcat thread keeps 
> listening to it.
> 
> There are always a maximum of 4 threads that keeps reading from
> the PHP Input Stream. As long as there are 4 "hanging" threads all
> other requests to tomcat work perfectly, which is very strange.
> 
> The thread dumps always show this:
> 
> java.lang.Thread.State: RUNNABLE at
> java.net.SocketInputStream.socketRead0(Native Method) at
> java.net.SocketInputStream.read(Unknown Source) at
> java.net.SocketInputStream.read(Unknown Source) at
> php.java.fastcgi.FCGIInputStream.read(FCGIInputStream.java:39) ...
> 
> At some point (up to 24 hours) tomcat "cleans" the threads and
> they stop. And after a while it begins with the failures until
> there are again 4 "hanging" threads.
> 
> If I look in VisualVM in the tab MBeans at the PhpCGIServlet of
> those processes, there is a maxTime of 3406, so PHP should not be
> the problem.

It's unclear to me whether the at
php.java.servlet.fastcgi.FastCGIServlet.doPost(FastCGIServlet.java:491)
method
is *handling* a POST -- that is, responding to a POST request to
Tomcat or *initiating* a POST to another service.  I suspect the
answer to the above question will help understand what is going on, here
.

I find it strange that there is code doing "PHP FastCGI" running
within a Java servlet container... what's wrong with regular HTTP?

- -chris


>> Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:
>>>>> From: Daniel Castilla | thin(k)design
>>>>> [mailto:d...@thin-k-design.com] Sent: Thursday, March 14,
>>>>> 2019 12:37 PM To: Jäkel, Guido ;
>>>>> users@tomcat.apache.org Subject: Re: Crash in http
>>>>> connector random once a day
>>>>> 
>>>>> Dear Guido,
>>>>> 
>>>>> thanks for the reply. The requests are reaching tomcat, and
>>>>> a thread is always started, if I look at the current
>>>>> threads on the tomcat manager I see the following, there
>>>>> are 4 threads that are processing since 2+ hours:
>>>>> 
>>>>> R ? ? ? ? ? ? S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1
>>>>> 0:0:0:0:0:0:0:1 localhost GET /manager/status HTTP/1.1 S
>>>>> 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1
>>>>> localhost POST /cloudworx/?method=words=17385 HTTP/1.1 S
>>>>> 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1
>>>>> localhost POST /cloudworx/?method=words=18986 HTTP/1.1 S
>>>>> 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1
>>>>> localhost POST /cloudworx/?method=words=10560 HTTP/1.1 R
>>>>> ? ? ? ? ? ? S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1
>>>>> 0:0:0:0:0:0:0:1 localhost POST
>>>>> >/cloudworx/?method=words=10560 HTTP/1.1
>>>>> 
>>>>> I'm not sure what other metrics would be helpful, but your
>>>>> Unix script wouldn't help much, as I am on a Windows Server
>>>>> 2012 and I would like to avoid installing Cygwin or
>>>>> something similar.
>>>> 
>>>> Dear Daniel,
>>>> 
>>>> the script just read-out the same core data and does some
>>>> pretty print. You may do it by your own by doing a
>>>> HTTP-Request against
>>>> 
>>>> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Cat
alina:type=RequestProcessor,*'
>>>>
>>>>
>>>>
>>>> 
What is interesting from your snippet that there are POST Request "in
Service" (i.e. Progress) since more than 2h. And you told that the ID is
unique, but there are two times a '10560'.
>>>> 
>>>> Is there a database service involved in the backend?
>>>> 
>>>> BTW: In addition to pull static thread dumps you may use
>>>> JVisualVM to get a live view to the threads.
>>> 
>>> Dear Guido and Mark,
>>> 
>>> the same Id '10560' appears twice in the URL as the same
>>> reque

Re: [OT] Crash in http connector random once a day

2019-03-15 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Daniel,

On 3/14/19 06:09, Daniel Castilla | thin(k)design wrote:
> Hi,
> 
> I have long searched and haven't found any real hint how to solve
> my problem, so I am adressing you.
> 
> Here are my specifications:
> 
> Server version:Apache Tomcat/8.5.15 Server built:
> May 5 2017 11:03:04 UTC Server number: 8.5.15.0 OS Name:
> Windows Server 2012 OS Version:6.2 Architecture:
> amd64 Java Home: C:\Program Files\Java\jre7 JVM
> Version:   1.7.0_25-b17 JVM Vendor:Oracle
> Corporation
> 
> The tomcat connector I am using has no custom configuration:
> 
> 
> 
> I have a PHP Javabridge app (runs PHP CGI to access JAVA
> functions) running on tomcat and it's beeing accessed solely
> through localhost from other PHP scripts running on IIS. It goes as
> follows:
> 
> 1. Request sent to IIS 2. Run PHP on IIS, process the request and
> delegates a part of the processing to tomcat 3. Access tomcat
> through HTTP on localhost:8080 and wait maximum 60s for the
> response, retry up to 3 times

I'm curious, if you are willing to wait for 3 minutes for a response
from the Tomcat server, why bother closing the request after 1 minute
and re-trying? That could potentially turn a single long-running
request into 3 simultaneous long-running requests, which may cause a
cascade and take-down lots other requests.

Why not simply wait up to 3 minutes a single time?

> In the JAVA logs I can see a "java.net.SocketException: Connection 
> reset", which is logical, as the PHP process on IIS closed the 
> connection after the 60s timeout. I tried to set this timeout to 
> infinite, with the result that the PHP process on IIS waited
> between 8 and 24 hours, only to get a 500 from tomcat.

So Tomcat will wait up to 24 hours before returning the 500 response?
That seems .. unusual.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyMQHAACgkQHPApP6U8
pFjRKBAAj2XN2+93wf3+eLxnBUvMSEoAmPEBqHRXAaIWRWxzUKLBZnq+BjjqccH6
Krlc9OemUIVAurwtOGuKx4erD+Iy2bCxAlit9NkeGJLKmPM1l90NXyglWNNqxZZD
1c5oXSm6Bmfws6Cw4Z0geEzRqHRWS6mBDl/W1XCmiK9wUMWmxFkol2xOrv74lMrO
m97ZK8uEEBFzFLqT4lF/n2SbP2A1BJjBfqCC44jBvAqlYN1DuXY/yIkjaIY/+Haw
ev0CJyjaICfu96QBNdQKvAnQTwHIUPy5SYBEQbgqyxXfiwbw7Gx+KXnSK0DK9N7b
Zs4VI93/R8y2+BhFZ8JjQjh0cJyJKxSl2TkL/CFkOdaWEFmRlqBZ6LDkujENFW6s
39x1Jni553+ISPe84nffCheIkGQDin/kP11cgRyJgmNyFTpXBKKTZkEy8zssJy28
Q2dLSMe3OIgpqVPOXeq/zb+uWwf/XOA4nBPVjY9MLuuZAedirpmHgzZgdCjNK/IT
eEZUjxFui56uvJTr4vQEf/dwMQVQB82KxezPaHD/qC3zldoVA86UWBWomwy/MeYb
Eaf+iHeOBZ2h4q+TJY3fy2wjd8xMHlLpPmtPY1+uBe6HzvuID/ytg9uoGMbWkXoe
w4Iaif3wPHItJld935Ccifi1UPhT/0N31GC6hVR+U7wbR9QlP2U=
=b7Gb
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-15 Thread Daniel Castilla | thin(k)design 
>
> In each of the four thread dumps, there are (the same) 4 threads
> connected to and waiting for an answer from a PHP FCGI backend behind
> Tomcat. You need to investigate, why that backend isn't sending a
> response (or takes so long for it).
>
> Regards,
>
> Rainer
>

Hi Rainer,

I have debugged PHP, the process ends normally and the response is
sent 100% of the time, so I don't know why the tomcat thread keeps
listening to it.

There are always a maximum of 4 threads that keeps reading from the
PHP Input Stream. As long as there are 4 "hanging" threads all other
requests to tomcat work perfectly, which is very strange.

The thread dumps always show this:

   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at php.java.fastcgi.FCGIInputStream.read(FCGIInputStream.java:39)
   ...

At some point (up to 24 hours) tomcat "cleans" the threads and they
stop. And after a while it begins with the failures until there are
again 4 "hanging" threads.

If I look in VisualVM in the tab MBeans at the PhpCGIServlet of those
processes, there is a maxTime of 3406, so PHP should not be the
problem.

A heapdump shows nothing unusual I think:

   Total Bytes: 35.381.250
   Total Classes: 6.655
   Total Instances: 454.066
   Classloaders: 178
   GC Roots: 3.318
   Number of Objects Pending for Finalization: 0

I am really out of ideas...





> Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:
> >>> From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >>> Sent: Thursday, March 14, 2019 12:37 PM
> >>> To: Jäkel, Guido ; users@tomcat.apache.org
> >>> Subject: Re: Crash in http connector random once a day
> >>>
> >>> Dear Guido,
> >>>
> >>> thanks for the reply. The requests are reaching tomcat, and a thread
> >>> is always started, if I look at the current threads on the tomcat
> >>> manager I see the following, there are 4 threads that are processing
> >>> since 2+ hours:
> >>>
> >>> R ? ? ? ? ? ?
> >>> S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET 
> >>> /manager/status HTTP/1.1
> >>> S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>> /cloudworx/?method=words=17385 HTTP/1.1
> >>> S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>> /cloudworx/?method=words=18986 HTTP/1.1
> >>> S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>> /cloudworx/?method=words=10560 HTTP/1.1
> >>> R ? ? ? ? ? ?
> >>> S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>> >/cloudworx/?method=words=10560 HTTP/1.1
> >>>
> >>> I'm not sure what other metrics would be helpful, but your Unix script
> >>> wouldn't help much, as I am on a Windows Server 2012 and I would like
> >>> to avoid installing Cygwin or something similar.
> >>
> >> Dear Daniel,
> >>
> >> the script just read-out the same core data and does some pretty print. 
> >> You may do it by your own by doing a HTTP-Request against
> >>
> >> 
> >> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
> >>
> >>
> >> What is interesting from your snippet that there are POST Request "in 
> >> Service" (i.e. Progress) since more than 2h. And you told that the ID is 
> >> unique, but there are two times a '10560'.
> >>
> >> Is there a database service involved in the backend?
> >>
> >> BTW: In addition to pull static thread dumps you may use JVisualVM to get 
> >> a live view to the threads.
> >
> > Dear Guido and Mark,
> >
> > the same Id '10560' appears twice in the URL as the same request was
> > retried for two times, the third time it worked.
> > The other two URLs had only one retry, the second time it worked.
> >
> > You can find a complete thread dump here (I hope the link gets
> > through): 
> > https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing
> >
> > I had to restart tomcat now in order to access the jmxproxy as it
> > wasn't active in the user roles, so the active "problem" processes are
> > gone for now. But it will happen again, at latest tomorrow.
> >
> > I tried JVisualVM, but I am no Java Expert and don't know what to search 
> > for...
> >
> > Thanks
> > Daniel

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-14 Thread Rainer Jung 
In each of the four thread dumps, there are (the same) 4 threads 
connected to and waiting for an answer from a PHP FCGI backend behind 
Tomcat. You need to investigate, why that backend isn't sending a 
response (or takes so long for it).


Regards,

Rainer

Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:

From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
Sent: Thursday, March 14, 2019 12:37 PM
To: Jäkel, Guido ; users@tomcat.apache.org
Subject: Re: Crash in http connector random once a day

Dear Guido,

thanks for the reply. The requests are reaching tomcat, and a thread
is always started, if I look at the current threads on the tomcat
manager I see the following, there are 4 threads that are processing
since 2+ hours:

R ? ? ? ? ? ?
S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET /manager/status 
HTTP/1.1
S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
/cloudworx/?method=words=17385 HTTP/1.1
S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
/cloudworx/?method=words=18986 HTTP/1.1
S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
/cloudworx/?method=words=10560 HTTP/1.1
R ? ? ? ? ? ?
S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=10560 HTTP/1.1

I'm not sure what other metrics would be helpful, but your Unix script
wouldn't help much, as I am on a Windows Server 2012 and I would like
to avoid installing Cygwin or something similar.


Dear Daniel,

the script just read-out the same core data and does some pretty print. You may 
do it by your own by doing a HTTP-Request against


URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'


What is interesting from your snippet that there are POST Request "in Service" 
(i.e. Progress) since more than 2h. And you told that the ID is unique, but there are two 
times a '10560'.

Is there a database service involved in the backend?

BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
live view to the threads.


Dear Guido and Mark,

the same Id '10560' appears twice in the URL as the same request was
retried for two times, the third time it worked.
The other two URLs had only one retry, the second time it worked.

You can find a complete thread dump here (I hope the link gets
through): 
https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing

I had to restart tomcat now in order to access the jmxproxy as it
wasn't active in the user roles, so the active "problem" processes are
gone for now. But it will happen again, at latest tomorrow.

I tried JVisualVM, but I am no Java Expert and don't know what to search for...

Thanks
Daniel


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
> Is there a database service involved in the backend?

No, the IDs are just for debugging, so I can identify every request uniquely.

Am Do., 14. März 2019 um 13:32 Uhr schrieb Jäkel, Guido :
>
> >From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >Sent: Thursday, March 14, 2019 12:37 PM
> >To: Jäkel, Guido ; users@tomcat.apache.org
> >Subject: Re: Crash in http connector random once a day
> >
> >Dear Guido,
> >
> >thanks for the reply. The requests are reaching tomcat, and a thread
> >is always started, if I look at the current threads on the tomcat
> >manager I see the following, there are 4 threads that are processing
> >since 2+ hours:
> >
> >R ? ? ? ? ? ?
> >S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET 
> >/manager/status HTTP/1.1
> >S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=17385 HTTP/1.1
> >S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=18986 HTTP/1.1
> >S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=10560 HTTP/1.1
> >R ? ? ? ? ? ?
> >S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>/cloudworx/?method=words=10560 HTTP/1.1
> >
> >I'm not sure what other metrics would be helpful, but your Unix script
> >wouldn't help much, as I am on a Windows Server 2012 and I would like
> >to avoid installing Cygwin or something similar.
>
> Dear Daniel,
>
> the script just read-out the same core data and does some pretty print. You 
> may do it by your own by doing a HTTP-Request against
>
>
> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
>
>
> What is interesting from your snippet that there are POST Request "in 
> Service" (i.e. Progress) since more than 2h. And you told that the ID is 
> unique, but there are two times a '10560'.
>
> Is there a database service involved in the backend?
>
> BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
> live view to the threads.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
> >From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >Sent: Thursday, March 14, 2019 12:37 PM
> >To: Jäkel, Guido ; users@tomcat.apache.org
> >Subject: Re: Crash in http connector random once a day
> >
> >Dear Guido,
> >
> >thanks for the reply. The requests are reaching tomcat, and a thread
> >is always started, if I look at the current threads on the tomcat
> >manager I see the following, there are 4 threads that are processing
> >since 2+ hours:
> >
> >R ? ? ? ? ? ?
> >S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET 
> >/manager/status HTTP/1.1
> >S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=17385 HTTP/1.1
> >S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=18986 HTTP/1.1
> >S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=10560 HTTP/1.1
> >R ? ? ? ? ? ?
> >S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>/cloudworx/?method=words=10560 HTTP/1.1
> >
> >I'm not sure what other metrics would be helpful, but your Unix script
> >wouldn't help much, as I am on a Windows Server 2012 and I would like
> >to avoid installing Cygwin or something similar.
>
> Dear Daniel,
>
> the script just read-out the same core data and does some pretty print. You 
> may do it by your own by doing a HTTP-Request against
>
>
> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
>
>
> What is interesting from your snippet that there are POST Request "in 
> Service" (i.e. Progress) since more than 2h. And you told that the ID is 
> unique, but there are two times a '10560'.
>
> Is there a database service involved in the backend?
>
> BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
> live view to the threads.

Dear Guido and Mark,

the same Id '10560' appears twice in the URL as the same request was
retried for two times, the third time it worked.
The other two URLs had only one retry, the second time it worked.

You can find a complete thread dump here (I hope the link gets
through): 
https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing

I had to restart tomcat now in order to access the jmxproxy as it
wasn't active in the user roles, so the active "problem" processes are
gone for now. But it will happen again, at latest tomorrow.

I tried JVisualVM, but I am no Java Expert and don't know what to search for...

Thanks
Daniel

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Crash in http connector random once a day

2019-03-14 Thread Jäkel , Guido 
>From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
>Sent: Thursday, March 14, 2019 12:37 PM
>To: Jäkel, Guido ; users@tomcat.apache.org
>Subject: Re: Crash in http connector random once a day
>
>Dear Guido,
>
>thanks for the reply. The requests are reaching tomcat, and a thread
>is always started, if I look at the current threads on the tomcat
>manager I see the following, there are 4 threads that are processing
>since 2+ hours:
>
>R ? ? ? ? ? ?
>S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET 
>/manager/status HTTP/1.1
>S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=17385 HTTP/1.1
>S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=18986 HTTP/1.1
>S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=10560 HTTP/1.1
>R ? ? ? ? ? ?
>S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>>/cloudworx/?method=words=10560 HTTP/1.1
>
>I'm not sure what other metrics would be helpful, but your Unix script
>wouldn't help much, as I am on a Windows Server 2012 and I would like
>to avoid installing Cygwin or something similar.

Dear Daniel,

the script just read-out the same core data and does some pretty print. You may 
do it by your own by doing a HTTP-Request against 

   
URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'


What is interesting from your snippet that there are POST Request "in Service" 
(i.e. Progress) since more than 2h. And you told that the ID is unique, but 
there are two times a '10560'.

Is there a database service involved in the backend?

BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
live view to the threads.

Re: Crash in http connector random once a day

2019-03-14 Thread Mark Thomas 
On 14/03/2019 11:37, Daniel Castilla | thin(k)design wrote:
> Dear Guido,
> 
> thanks for the reply. The requests are reaching tomcat, and a thread
> is always started, if I look at the current threads on the tomcat
> manager I see the following, there are 4 threads that are processing
> since 2+ hours:

When you see this take 3 Java thread dumps ~10s apart and post them
somewhere we can look at them. We want to find out what those threads
are doing.

Mark


> 
> R ? ? ? ? ? ?
> S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET
> /manager/status HTTP/1.1
> S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=17385 HTTP/1.1
> S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=18986 HTTP/1.1
> S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=10560 HTTP/1.1
> R ? ? ? ? ? ?
> S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=10560 HTTP/1.1
> 
> I'm not sure what other metrics would be helpful, but your Unix script
> wouldn't help much, as I am on a Windows Server 2012 and I would like
> to avoid installing Cygwin or something similar.
> 
> Regards,
> Daniel
> 
> 
> Am Do., 14. März 2019 um 12:02 Uhr schrieb Jäkel, Guido :
>>
>> Dear Daniel,
>>
>> a request is logged in the access log after it has finished. (In addition, 
>> on Tomcat the log is flushed with some delay, but that's not the problem 
>> here).
>>
>> But if the request is stall while processing, there's no hint in the access 
>> log. Therefore: Is there an application log that may confirm that the 
>> request have reached the application on Tomcat and have start to process?
>>
>> On Tomcat, there's also a request scoreboard feature where you may "live 
>> watch" outstanding requests. You may access to it via the Tomcat Host 
>> Manager Application. As a hint, I may also provide you a short (Unix) script 
>> to readout this from the MBean by HTTP via the jmxproxy-Feature of the 
>> Manager.
>>
>> Greetings
>>
>> Guido
>>
>>> -Original Message-
>>> From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
>>> Sent: Thursday, March 14, 2019 11:10 AM
>>> To: users@tomcat.apache.org
>>> Subject: Crash in http connector random once a day
>>>
>>> The strange thing is, those failed requests with no response are
>>> logged in the tomcat access logs with a 500 http connection error all
>>> at the same time (although they begun with 1-2 min difference) and
>>> after 8 to 24 hours (the URL is alway unique, so I know for sure).
>>
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
Dear Guido,

thanks for the reply. The requests are reaching tomcat, and a thread
is always started, if I look at the current threads on the tomcat
manager I see the following, there are 4 threads that are processing
since 2+ hours:

R ? ? ? ? ? ?
S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET
/manager/status HTTP/1.1
S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=17385 HTTP/1.1
S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=18986 HTTP/1.1
S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=10560 HTTP/1.1
R ? ? ? ? ? ?
S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=10560 HTTP/1.1

I'm not sure what other metrics would be helpful, but your Unix script
wouldn't help much, as I am on a Windows Server 2012 and I would like
to avoid installing Cygwin or something similar.

Regards,
Daniel


Am Do., 14. März 2019 um 12:02 Uhr schrieb Jäkel, Guido :
>
> Dear Daniel,
>
> a request is logged in the access log after it has finished. (In addition, on 
> Tomcat the log is flushed with some delay, but that's not the problem here).
>
> But if the request is stall while processing, there's no hint in the access 
> log. Therefore: Is there an application log that may confirm that the request 
> have reached the application on Tomcat and have start to process?
>
> On Tomcat, there's also a request scoreboard feature where you may "live 
> watch" outstanding requests. You may access to it via the Tomcat Host Manager 
> Application. As a hint, I may also provide you a short (Unix) script to 
> readout this from the MBean by HTTP via the jmxproxy-Feature of the Manager.
>
> Greetings
>
> Guido
>
> >-Original Message-
> >From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >Sent: Thursday, March 14, 2019 11:10 AM
> >To: users@tomcat.apache.org
> >Subject: Crash in http connector random once a day
> >
> >The strange thing is, those failed requests with no response are
> >logged in the tomcat access logs with a 500 http connection error all
> >at the same time (although they begun with 1-2 min difference) and
> >after 8 to 24 hours (the URL is alway unique, so I know for sure).
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Crash in http connector random once a day

2019-03-14 Thread Jäkel , Guido 
Dear Daniel,

a request is logged in the access log after it has finished. (In addition, on 
Tomcat the log is flushed with some delay, but that's not the problem here).

But if the request is stall while processing, there's no hint in the access 
log. Therefore: Is there an application log that may confirm that the request 
have reached the application on Tomcat and have start to process?

On Tomcat, there's also a request scoreboard feature where you may "live watch" 
outstanding requests. You may access to it via the Tomcat Host Manager 
Application. As a hint, I may also provide you a short (Unix) script to readout 
this from the MBean by HTTP via the jmxproxy-Feature of the Manager.

Greetings

Guido

>-Original Message-
>From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
>Sent: Thursday, March 14, 2019 11:10 AM
>To: users@tomcat.apache.org
>Subject: Crash in http connector random once a day
>
>The strange thing is, those failed requests with no response are
>logged in the tomcat access logs with a 500 http connection error all
>at the same time (although they begun with 1-2 min difference) and
>after 8 to 24 hours (the URL is alway unique, so I know for sure).

Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
Hi,

I have long searched and haven't found any real hint how to solve my
problem, so I am adressing you.

Here are my specifications:

Server version:Apache Tomcat/8.5.15
Server built:  May 5 2017 11:03:04 UTC
Server number: 8.5.15.0
OS Name:   Windows Server 2012
OS Version:6.2
Architecture:  amd64
Java Home: C:\Program Files\Java\jre7
JVM Version:   1.7.0_25-b17
JVM Vendor:Oracle Corporation

The tomcat connector I am using has no custom configuration:



I have a PHP Javabridge app (runs PHP CGI to access JAVA functions)
running on tomcat and it's beeing accessed solely through localhost
from other PHP scripts running on IIS. It goes as follows:

1. Request sent to IIS
2. Run PHP on IIS, process the request and delegates a part of the
processing to tomcat
3. Access tomcat through HTTP on localhost:8080 and wait maximum 60s
for the response, retry up to 3 times
4. Run PHP on tomcat, process the request and send data back to the
original PHP process on IIS (on 2.)
5. On IIS read the response from tomcat, process it and send a
response to the original request (on 1.)

For 99% of the time it goes well, we have around 500+ requests beeing
succesfully processed. BUT aprox. once a day there are around 4
requests to tomcat within 3 minutes or so that aren't receiving any
response at all (I have setup a 60s timeout in the IIS PHP process,
usually the tomcat process takes up to 10s to respond).

The strange thing is, those failed requests with no response are
logged in the tomcat access logs with a 500 http connection error all
at the same time (although they begun with 1-2 min difference) and
after 8 to 24 hours (the URL is alway unique, so I know for sure).

In the JAVA logs I can see a "java.net.SocketException: Connection
reset", which is logical, as the PHP process on IIS closed the
connection after the 60s timeout. I tried to set this timeout to
infinite, with the result that the PHP process on IIS waited between 8
and 24 hours, only to get a 500 from tomcat. This time I got in the
JAVA logs a "java.net.SocketException: Connection reset by peer"
(notice the difference "by peer").

I hope I described my problem accurate enough. I'm out of ideas and
I'm grateful for any insight!

Regards,
Daniel

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
Hi,

I have long searched and haven't found any real hint how to solve my
problem, so I am adressing you.

Here are my specifications:

Server version:Apache Tomcat/8.5.15
Server built:  May 5 2017 11:03:04 UTC
Server number: 8.5.15.0
OS Name:   Windows Server 2012
OS Version:6.2
Architecture:  amd64
Java Home: C:\Program Files\Java\jre7
JVM Version:   1.7.0_25-b17
JVM Vendor:Oracle Corporation

The tomcat connector I am using has no custom configuration:



I have a PHP Javabridge app (runs PHP CGI to access JAVA functions)
running on tomcat and it's beeing accessed solely through localhost
from other PHP scripts running on IIS. It goes as follows:

1. Request sent to IIS
2. Run PHP on IIS, process the request and delegates a part of the
processing to tomcat
3. Access tomcat through HTTP on localhost:8080 and wait maximum 60s
for the response, retry up to 3 times
4. Run PHP on tomcat, process the request and send data back to the
original PHP process on IIS (on 2.)
5. On IIS read the response from tomcat, process it and send a
response to the original request (on 1.)

For 99% of the time it goes well, we have around 500+ requests beeing
succesfully processed. BUT aprox. once a day there are around 4
requests to tomcat within 3 minutes or so that aren't receiving any
response at all (I have setup a 60s timeout in the IIS PHP process,
usually the tomcat process takes up to 10s to respond).

The strange thing is, those failed requests with no response are
logged in the tomcat access logs with a 500 http connection error all
at the same time (although they begun with 1-2 min difference) and
after 8 to 24 hours (the URL is alway unique, so I know for sure).

In the JAVA logs I can see a "java.net.SocketException: Connection
reset", which is logical, as the PHP process on IIS closed the
connection after the 60s timeout. I tried to set this timeout to
infinite, with the result that the PHP process on IIS waited between 8
and 24 hours, only to get a 500 from tomcat. This time I got in the
JAVA logs a "java.net.SocketException: Connection reset by peer"
(notice the difference "by peer").

I hope I described my problem accurate enough. I'm out of ideas and
I'm grateful for any insight!

Regards,
Daniel

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org