Re: TLS protocols and cipher suites available under JSSE?

2019-03-14 Thread James H. H. Lampert 

On 3/13/19, 2:53 PM, Christopher Schultz wrote:


7: 
https://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html

8: 
https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html

If you are using a non-standard "provider" (like one from IBM!), then
you'll have to read their documentation to find out what they provide.


Dear Mr. Schultz:

I was able to do a bit of trial-and-error this afternoon.

I found that on an AS/400 running Tomcat 7.0.47 under Java 7, with IBM 
JSSE (see 
 
for reference)


If I specified

sslProtocol="TLSv1.1,TLSv1.2"


then connector initialization blew up.

If I specified

sslProtocol="TLSv1.2"


then connector initialization didn't blow up, but neither did I get TLSv1.2.

If I specified

sslEnabledProtocols="TLSv1.1,TLSv1.2" sslProtocol="TLSv1.1,TLSv1.2"


it blew up again. Apparently it doesn't like a comma-separated list for 
sslProtocol. This is what showed up in catalina.out:

SEVERE: Failed to initialize end point associated with ProtocolHandler 
["http-bio-12.151.5.120-443"]
java.io.IOException: TLSv1.1,TLSv1.2 SSLContext not available
at 
org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:459)
at 
org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:192)
at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:397)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:623)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:434)
at 
org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
. . .
Caused by: java.security.NoSuchAlgorithmException: TLSv1.1,TLSv1.2 SSLContext 
not available
at sun.security.jca.GetInstance.getInstance(GetInstance.java:172)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:26)
at 
org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSSLContext(JSSESocketFactory.java:472)
at 
org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:433)

. . .

But if I do

sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" sslProtocol="SSL_TLSv2"


(with "SSL_TLSv2" being an IBM special value for "anything that's 
enabled, up to and including TLSv1.2), then it works, and indicates a 
TLSv1.2 connection, from a reasonably current version of Chrome, and 
also works from a very old Firefox that doesn't go past TLSv1.0.


And if I tweak the "sslEnabledProtocols," value, I can get it to reject 
1.0, or reject 1.2, in predictable ways.


--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

2 instance of ParallelWebAppClassLoaders in Tomcat JVM

2019-03-14 Thread Shankar P S 
Version - Tomcat 8.5.38
OS - Centos 7.6
JVM - openjdk 1.8, 4 CPU

We recently upgraded an existing server running a Spring 3.0 web app from
Tomcat 8.5.35 to 8.5.38.

After the upgrade, I noticed OutOfMemory exceptions several times.
Increasing the heap size (-Xmx) several times upto nearly twice (from 6 GB
to 10 GB) helped bring the problem under control.

I was worried about a memory leak and so took a heap dump. Upon inspection
I noticed that there were 2 copies of 2 large HashMap objects that we use
as a cache for fast lookup. I noticed that this was because there were 2
instances of org.apache.catalina.loader.ParallelWebappClassLoader.

Downgrading to 5.3.35, reduced the heap size to half and the problem was
gone. In the new heap dump, there was only one instance of
ParallelWebappClassLoader.

My question is - why does Tomcat 5.3.38 create 2 instances of
ParallelWebappClassLoader, while  5.3.35 creates only one?Is there a way to
conigure the number of instances of  ParallelWebappClassLoaderwhen
upgrading to 8.5.38?

Thanks for reading.

I have also posted the same problem in Stackoverflow -
https://stackoverflow.com/questions/55154963/behavior-of-multiple-instance-of-parallelwebappclassloaders-in-tomcat-jvm-retain

Tomcat doesn't restart

2019-03-14 Thread Louis Zipes 
Hi Experts,
Running Tomcat 7.0.54 on Windows 2012

Our morning process is to fire a stop of the Tomcat Windows Service and then an 
immediate restart.  This is more due to the 3rd party application that is 
running with Tomcat than a problem with Tomcat.  A few weeks ago it started to 
not cleanly stop when using the following batch file command.  Note that this 
had worked for more than a year with no problems and I don't think that there 
have been any server changes although I don't control the machine


net stop "Apache Tomcat 7.0 TPMDEV" && net start "Apache Tomcat 7.0 TPMDEV"

I then changed the batch file, that calls the Windows Service stop/start to 
give a 65 second delay between the stop and start hoping to give it more time 
to cleanly stop before restarting:




net stop "Apache Tomcat 7.0 TPMDEV"

PING localhost -n 65 >NUL

net start "Apache Tomcat 7.0 TPMDEV"




This modified batch file worked for about 2 weeks but then this morning it 
failed to stop and restart correctly.  Since it is PRD we just have to get it 
restarted so I usually don't have time to troubleshoot getting threads, etc.  
(also, the off shore team is on shift when this happens but I'm the 
administrator that actually has to get to root cause when I start my shift).

I observed the following in the std-output log it seems to receive the Shutdown 
command but it seems like either all of the threads are not being closed for 
some reason

Mar 14, 2019 5:28:00 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the 
Server instance.
Mar 14, 2019 5:28:00 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-7005"]
Mar 14, 2019 5:28:00 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Mar 14, 2019 5:28:00 AM org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 94 instance(s) to be deallocated for Servlet 
[RequestDispatcherServlet]
Mar 14, 2019 5:28:01 AM org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 94 instance(s) to be deallocated for Servlet 
[RequestDispatcherServlet]
Mar 14, 2019 5:28:03 AM org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 94 instance(s) to be deallocated for Servlet 
[RequestDispatcherServlet]
--


So my questions are:

1) Does anyone have a better way to make sure Tomcat is fully stopped and any 
open threads are killed
2) Is this part below real cause and I need to find the root cause by doing a 
thread dump?  As mentioned it is hard for me to get one because I'm usually not 
on shift when this happens.

Mar 14, 2019 5:28:03 AM org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 94 instance(s) to be deallocated for Servlet 
[RequestDispatcherServlet]

Thanks, Louis

---
CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and may 
contain information that is confidential, proprietary or exempt from 
disclosure. If you are not the intended recipient, please contact the sender 
immediately. Unauthorized use or distribution is prohibited and may be unlawful.

Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload

2019-03-14 Thread Rémy Maucherat 
On Thu, Mar 14, 2019 at 2:58 PM Santhosh Kumar 
wrote:

> I have tried the same set of testcases using h2load with tomcat 9.0.17
> from github releases
> (https://github.com/apache/tomcat/releases/tag/9.0.17) .
>
> h2load -n 100 -d /tmp/1k https://localhost:8443/
> requests: 100 total, 64 started, 63 done, 63 succeeded, 37 failed, 37
> errored, 0 timeout
>
> h2load -n 200 -d /tmp/512b https://localhost:8443/
> requests: 200 total, 128 started, 127 done, 127 succeeded, 73 failed,
> 73 errored, 0 timeout
>
> The issue still occurs in latest version too.
>

This is not related to the issue that you reported (you gave a stacktrace
for it, and it was helpful). Please use a real test with a real servlet
consuming the data,

Rémy

Re: Crash in http connector random once a day

2019-03-14 Thread Rainer Jung 
In each of the four thread dumps, there are (the same) 4 threads 
connected to and waiting for an answer from a PHP FCGI backend behind 
Tomcat. You need to investigate, why that backend isn't sending a 
response (or takes so long for it).


Regards,

Rainer

Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:

From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
Sent: Thursday, March 14, 2019 12:37 PM
To: Jäkel, Guido ; users@tomcat.apache.org
Subject: Re: Crash in http connector random once a day

Dear Guido,

thanks for the reply. The requests are reaching tomcat, and a thread
is always started, if I look at the current threads on the tomcat
manager I see the following, there are 4 threads that are processing
since 2+ hours:

R ? ? ? ? ? ?
S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET /manager/status 
HTTP/1.1
S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
/cloudworx/?method=words=17385 HTTP/1.1
S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
/cloudworx/?method=words=18986 HTTP/1.1
S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
/cloudworx/?method=words=10560 HTTP/1.1
R ? ? ? ? ? ?
S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=10560 HTTP/1.1

I'm not sure what other metrics would be helpful, but your Unix script
wouldn't help much, as I am on a Windows Server 2012 and I would like
to avoid installing Cygwin or something similar.


Dear Daniel,

the script just read-out the same core data and does some pretty print. You may 
do it by your own by doing a HTTP-Request against


URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'


What is interesting from your snippet that there are POST Request "in Service" 
(i.e. Progress) since more than 2h. And you told that the ID is unique, but there are two 
times a '10560'.

Is there a database service involved in the backend?

BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
live view to the threads.


Dear Guido and Mark,

the same Id '10560' appears twice in the URL as the same request was
retried for two times, the third time it worked.
The other two URLs had only one retry, the second time it worked.

You can find a complete thread dump here (I hope the link gets
through): 
https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing

I had to restart tomcat now in order to access the jmxproxy as it
wasn't active in the user roles, so the active "problem" processes are
gone for now. But it will happen again, at latest tomorrow.

I tried JVisualVM, but I am no Java Expert and don't know what to search for...

Thanks
Daniel


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
> Is there a database service involved in the backend?

No, the IDs are just for debugging, so I can identify every request uniquely.

Am Do., 14. März 2019 um 13:32 Uhr schrieb Jäkel, Guido :
>
> >From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >Sent: Thursday, March 14, 2019 12:37 PM
> >To: Jäkel, Guido ; users@tomcat.apache.org
> >Subject: Re: Crash in http connector random once a day
> >
> >Dear Guido,
> >
> >thanks for the reply. The requests are reaching tomcat, and a thread
> >is always started, if I look at the current threads on the tomcat
> >manager I see the following, there are 4 threads that are processing
> >since 2+ hours:
> >
> >R ? ? ? ? ? ?
> >S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET 
> >/manager/status HTTP/1.1
> >S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=17385 HTTP/1.1
> >S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=18986 HTTP/1.1
> >S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=10560 HTTP/1.1
> >R ? ? ? ? ? ?
> >S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>/cloudworx/?method=words=10560 HTTP/1.1
> >
> >I'm not sure what other metrics would be helpful, but your Unix script
> >wouldn't help much, as I am on a Windows Server 2012 and I would like
> >to avoid installing Cygwin or something similar.
>
> Dear Daniel,
>
> the script just read-out the same core data and does some pretty print. You 
> may do it by your own by doing a HTTP-Request against
>
>
> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
>
>
> What is interesting from your snippet that there are POST Request "in 
> Service" (i.e. Progress) since more than 2h. And you told that the ID is 
> unique, but there are two times a '10560'.
>
> Is there a database service involved in the backend?
>
> BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
> live view to the threads.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: HTTPS Invalid character found in method name. HTTP method names must be tokens.

2019-03-14 Thread Mark Thomas 
On 13/03/2019 14:41, Jan Vomlel wrote:
> We use selenium for our application testing. Our tests sometime fail
> with message "Invalid character found in method name" Error occures
> only on https and on on firefox 60 and internet explorer 11. Chrome,
> edge is OK.
> 
> We use Tomcat 9.0.16, Java 11 (Adopt Open JDK 11.0.2+9) on Linux,
> browsers are on windows 10.
> 
> We think, that there must be some error in https implementation. Log in
> these situations always contains request with application_data and
> change_cipher_spec together. But we do not understand https in these
> details.

That sounds like the previous request did not complete correctly leading
to the next request not being started at the correct point. You can try
enabling debug logging for:

org.apache.coyote.http11.Http11InputBuffer

That should tell you what request lines are being parsed.

Mark

> 
> Thanks for any advice, Jan Vomlel
> 
> javax.net.ssl|DEBUG|37|https-jsse-nio-8444-exec-6|2019-02-27
> 18:03:16.326 CET|SSLEngineInputRecord.java:177|Raw read (
>   : 17 03 03 02 74 00 00 00   00 00 00 00 01 E5 6A 79 t.jy
>   0010: CF D2 7A 6E 53 FB B3 97   3B 82 92 E5 7B A8 A2 EA ..znS...;...
>   0020: 4B B5 70 11 DE CD 7E 8C   89 08 AD 67 47 82 E1 16 K.pgG...
>   0030: FE 09 9A 1B F6 77 6C 67   80 0E CA 5F 55 4E 2C 2D .wlg..._UN,-
>   0040: D8 7B D2 71 2E 66 B4 0A   DA 8D 8F 11 C6 C3 27 1B ...q.f'.
>   0050: 18 82 16 FE 82 7C 83 B4   3B 43 D8 81 71 9E 27 22 ;C..q.'"
>   0060: 76 50 EB C6 4C 11 C1 BE   01 8E B9 6A 3A 0B 6C 6F vP..L..j:.lo
>   0070: 01 03 74 F1 C4 90 C7 52   A6 8D 4A A8 8D AC EF A0 ..tR..J.
>   0080: 62 03 3D C7 6E F9 FB 39   C5 FA A6 95 FD 46 C3 51 b.=.n..9.F.Q
>   0090: FE 67 2E 76 44 7B B1 B6   8C 34 F4 30 EC 93 EC 1D .g.vD4.0
>   00A0: A1 5B 01 2B C1 DA D3 AA   88 EC E8 31 66 5F 59 CA .[.+...1f_Y.
>   00B0: 38 9A 53 C5 89 31 FB FF   1D 59 6D 90 08 66 DB 6C 8.S..1...Ym..f.l
>   00C0: 6F 4A 9C F7 3A BE D8 5D   5C 3C AA 3E 2B A5 A8 E2 oJ..:..]\<.>+...
>   00D0: 54 50 65 7B 9A BA 92 71   0F 7B AA 58 DF B2 AC 3E TPeq...X...>
>   00E0: 5B 4E A1 29 9C F2 C6 1A   5E 6B 6A 85 19 DE 1C 73 [N.)^kjs
>   00F0: EF D2 AC 06 48 50 8D DD   66 F7 78 87 50 00 28 26 HP..f.x.P.(&
>   0100: FB A7 C1 87 30 67 5B FA   C8 B5 C7 41 4A 27 8E 6D 0g[AJ'.m
>   0110: D8 99 89 BA 32 8A 94 7F   79 2D 66 53 8D F4 6E 82 2...y-fS..n.
>   0120: 95 22 FF C6 E1 5E 8F B2   AA 44 25 33 14 F4 35 A1 ."...^...D%3..5.
>   0130: 50 84 F5 EE 5D 66 B6 AC   5A 21 72 5B 56 A3 32 07 P...]f..Z!r[V.2.
>   0140: 83 DF 3B A4 85 A9 3F E9   7B E6 4F 7C 73 F9 87 F5 ..;...?...O.s...
>   0150: 16 83 E7 24 CD 50 41 F9   43 37 CD 9D DF BE 69 E4 ...$.PA.C7i.
>   0160: AE 2D 81 BC 24 81 04 42   72 CE AA 76 5D 63 B4 98 .-..$..Br..v]c..
>   0170: A7 B4 AB 77 F2 6C 42 4A   75 04 06 BB 19 5A 19 A5 ...w.lBJuZ..
>   0180: CA E6 FE 04 61 8B 90 B4   E5 D1 B5 9D C1 49 27 5C aI'\
>   0190: 41 92 DE 7B 91 AB 7D 3B   21 B9 60 A5 A2 C7 92 19 A..;!.`.
>   01A0: 2F 3D 82 17 FB C0 20 7A   E6 79 7C B6 2E CD 50 A9  /= z.yP.
>   01B0: 6B 8E 7E 04 E9 32 EC 29   22 EC BB 58 C5 5F 8C A6 k2.)"..X._..
>   01C0: 1D 76 CB 4E 5A 56 C7 83   C2 74 AC D5 83 70 05 80 .v.NZV...t...p..
>   01D0: 19 8B E8 CF 09 98 32 D1   8B 4A 93 7B 38 F0 8A 7B ..2..J..8...
>   01E0: AB 88 D5 99 E3 42 76 1B   C9 EB 62 94 0C F6 DA EF .Bv...b.
>   01F0: 9F 4D DB F0 30 CA 34 13   E7 1A 07 39 FD 92 F5 08 .M..0.49
>   0200: F4 E5 62 26 94 BF 3C 6A   82 4D 06 C8 96 57 12 F3 ..b&..   0210: 07 03 09 0B 04 F9 C6 0E   D2 AD 85 DC C4 09 2D 5A ..-Z
>   0220: 44 0D 37 42 57 44 C6 05   F6 F7 4E 9E A7 79 BC F3 D.7BWDN..y..
>   0230: 94 F5 67 96 B9 6C 2D 9E   E2 C4 77 23 11 20 F7 4A ..g..l-...w#. .J
>   0240: AE 73 B3 28 20 C7 1F 92   99 C8 55 A3 18 24 85 54  .s.( .U..$.T
>   0250: 19 90 1F 24 73 91 A7 3F   15 BA 83 D3 39 03 8C 89 ...$s..?9...
>   0260: D5 70 72 97 AF 91 4B 11   3B 72 E4 91 B5 6D BC 29 .pr...K.;r...m.)
>   0270: 51 C6 47 5C 73 58 84 A7   26 14 03 03 00 01 01 16 Q.G\sX..&...
>   0280: 03 03 00 28 00 00 00 00   00 00 00 00 17 E8 48 1B ...(..H.
>   0290: 07 8D 8A E9 98 CA 25 1F   AD D5 02 FC 0A C9 8E 4D ..%M
>   02A0: F6 C6 EA 2E D6 24 8C D0   11 DA 78 D3 .$x.
> )
> javax.net.ssl|DEBUG|37|https-jsse-nio-8444-exec-6|2019-02-27
> 18:03:16.326 CET|SSLEngineInputRecord.java:214|READ: TLSv1.2
> application_data, length = 628
> javax.net.ssl|DEBUG|37|https-jsse-nio-8444-exec-6|2019-02-27
> 18:03:16.326 CET|SSLEngineInputRecord.java:177|Raw read (
>   : 14 03 03 00 01 01 16 03   03 00 28 00 00 00 00 00 ..(.
>   0010: 00 00 00 17 E8 48 1B 07   8D 8A E9 98 CA 25 1F AD .H...%..
>   0020: D5 02 FC 0A C9 8E 4D F6   C6 EA 2E D6 24 8C D0 11 ..M.$...
>   0030: DA 78 D3   .x.
> )
> javax.net.ssl|DEBUG|37|https-jsse-nio-8444-exec-6|2019-02-27
> 18:03:16.326

Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload

2019-03-14 Thread Santhosh Kumar 
I have tried the same set of testcases using h2load with tomcat 9.0.17
from github releases
(https://github.com/apache/tomcat/releases/tag/9.0.17) .

h2load -n 100 -d /tmp/1k https://localhost:8443/
requests: 100 total, 64 started, 63 done, 63 succeeded, 37 failed, 37
errored, 0 timeout

h2load -n 200 -d /tmp/512b https://localhost:8443/
requests: 200 total, 128 started, 127 done, 127 succeeded, 73 failed,
73 errored, 0 timeout

The issue still occurs in latest version too.


On Fri, Mar 8, 2019 at 1:23 AM Rémy Maucherat  wrote:

> On Thu, Mar 7, 2019 at 5:22 PM Rémy Maucherat  wrote:
>
> > On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat  wrote:
> >
> >> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas  wrote:
> >>
> >>> On 07/03/2019 07:40, Santhosh Kumar wrote:
> >>> > From some of the test cases I can safely say that tomcat is hitting
> >>> some
> >>> > limits, I have two test cases ran with two diff size of payload and
> >>> without
> >>> > any queryParams. The servlet is a empty servlet just returns after
> >>> > receiving without doing any business side logic
> >>>
> >>> Can you repeat those tests with the NIO connector? It would be helpful
> >>> to know if we should be looking at the HTTP/2 code or the low-level
> >>> connector I/O code.
> >>>
> >>
> >> I was planning to investigate since I'm hunting NIO2 additional issues
> >> after the fix for BZ63182. This one looks simpler to reproduce at least
> >> [assuming there's an issue].
> >>
> >
> > Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used
> > are too optimized.
> >
>
> The "fix" will be in Tomcat 9.0.17, the read buffer used simply needs to be
> larger with SSL. Both the JSSE and OpenSSL engines exhibit the same
> behavior, so no possible workaround elsewhere.
>
> Rémy
>


-- 
*With Regards,*
*Santhosh Kumar J*

Re: Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
> >From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >Sent: Thursday, March 14, 2019 12:37 PM
> >To: Jäkel, Guido ; users@tomcat.apache.org
> >Subject: Re: Crash in http connector random once a day
> >
> >Dear Guido,
> >
> >thanks for the reply. The requests are reaching tomcat, and a thread
> >is always started, if I look at the current threads on the tomcat
> >manager I see the following, there are 4 threads that are processing
> >since 2+ hours:
> >
> >R ? ? ? ? ? ?
> >S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET 
> >/manager/status HTTP/1.1
> >S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=17385 HTTP/1.1
> >S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=18986 HTTP/1.1
> >S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >/cloudworx/?method=words=10560 HTTP/1.1
> >R ? ? ? ? ? ?
> >S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
> >>/cloudworx/?method=words=10560 HTTP/1.1
> >
> >I'm not sure what other metrics would be helpful, but your Unix script
> >wouldn't help much, as I am on a Windows Server 2012 and I would like
> >to avoid installing Cygwin or something similar.
>
> Dear Daniel,
>
> the script just read-out the same core data and does some pretty print. You 
> may do it by your own by doing a HTTP-Request against
>
>
> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
>
>
> What is interesting from your snippet that there are POST Request "in 
> Service" (i.e. Progress) since more than 2h. And you told that the ID is 
> unique, but there are two times a '10560'.
>
> Is there a database service involved in the backend?
>
> BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
> live view to the threads.

Dear Guido and Mark,

the same Id '10560' appears twice in the URL as the same request was
retried for two times, the third time it worked.
The other two URLs had only one retry, the second time it worked.

You can find a complete thread dump here (I hope the link gets
through): 
https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing

I had to restart tomcat now in order to access the jmxproxy as it
wasn't active in the user roles, so the active "problem" processes are
gone for now. But it will happen again, at latest tomorrow.

I tried JVisualVM, but I am no Java Expert and don't know what to search for...

Thanks
Daniel

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Crash in http connector random once a day

2019-03-14 Thread Jäkel , Guido 
>From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
>Sent: Thursday, March 14, 2019 12:37 PM
>To: Jäkel, Guido ; users@tomcat.apache.org
>Subject: Re: Crash in http connector random once a day
>
>Dear Guido,
>
>thanks for the reply. The requests are reaching tomcat, and a thread
>is always started, if I look at the current threads on the tomcat
>manager I see the following, there are 4 threads that are processing
>since 2+ hours:
>
>R ? ? ? ? ? ?
>S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET 
>/manager/status HTTP/1.1
>S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=17385 HTTP/1.1
>S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=18986 HTTP/1.1
>S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>/cloudworx/?method=words=10560 HTTP/1.1
>R ? ? ? ? ? ?
>S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST 
>>/cloudworx/?method=words=10560 HTTP/1.1
>
>I'm not sure what other metrics would be helpful, but your Unix script
>wouldn't help much, as I am on a Windows Server 2012 and I would like
>to avoid installing Cygwin or something similar.

Dear Daniel,

the script just read-out the same core data and does some pretty print. You may 
do it by your own by doing a HTTP-Request against 

   
URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'


What is interesting from your snippet that there are POST Request "in Service" 
(i.e. Progress) since more than 2h. And you told that the ID is unique, but 
there are two times a '10560'.

Is there a database service involved in the backend?

BTW: In addition to pull static thread dumps you may use JVisualVM to get a 
live view to the threads.

Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)

2019-03-14 Thread Igor T 
9.0.17 was used from
http://tomcat.10.x6.nabble.com/VOTE-Release-Apache-Tomcat-9-0-17-td5083815.html

I've created a new totally clean AWS instance with new domain name and
new Lets Encrypt certificates.
It's available here:
 temp-test-01.tk
Configuration:
OS: Windows Server 2012 R2 Base
Tomcat: totally new distribution of 9.0.16 windows x64 build
Java: Oracle jdk1.8.0_181 (you can download it from here
https://temp-test-01.tk/jdk1.8.0_181.7z)
Tests were done from multiple locations to exclude network devices problem.

Running the test with the same jdk1.8.0_181 on local PC, getting the
same resets.

Running the test with the jdk1.8.0_144 or latest jdk1.8.0_202 on local
PC, getting even worse (first socket gets reply and all others getting
timeout):
success: 1, read 196 bytes for: 88ms
success: 2, read -1 bytes for: 5234ms
success: 3, read -1 bytes for: 5231ms
success: 4, read -1 bytes for: 5247ms

success: 1, read 196 bytes for: 110ms
success: 2, read -1 bytes for: 5419ms
success: 3, read -1 bytes for: 5232ms
success: 4, read -1 bytes for: 5232ms





Full tomcat server.xml:



  
  
  
  
  

  

  

  













  

  

  



  

  


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-14 Thread Mark Thomas 
On 14/03/2019 11:37, Daniel Castilla | thin(k)design wrote:
> Dear Guido,
> 
> thanks for the reply. The requests are reaching tomcat, and a thread
> is always started, if I look at the current threads on the tomcat
> manager I see the following, there are 4 threads that are processing
> since 2+ hours:

When you see this take 3 Java thread dumps ~10s apart and post them
somewhere we can look at them. We want to find out what those threads
are doing.

Mark


> 
> R ? ? ? ? ? ?
> S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET
> /manager/status HTTP/1.1
> S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=17385 HTTP/1.1
> S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=18986 HTTP/1.1
> S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=10560 HTTP/1.1
> R ? ? ? ? ? ?
> S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> /cloudworx/?method=words=10560 HTTP/1.1
> 
> I'm not sure what other metrics would be helpful, but your Unix script
> wouldn't help much, as I am on a Windows Server 2012 and I would like
> to avoid installing Cygwin or something similar.
> 
> Regards,
> Daniel
> 
> 
> Am Do., 14. März 2019 um 12:02 Uhr schrieb Jäkel, Guido :
>>
>> Dear Daniel,
>>
>> a request is logged in the access log after it has finished. (In addition, 
>> on Tomcat the log is flushed with some delay, but that's not the problem 
>> here).
>>
>> But if the request is stall while processing, there's no hint in the access 
>> log. Therefore: Is there an application log that may confirm that the 
>> request have reached the application on Tomcat and have start to process?
>>
>> On Tomcat, there's also a request scoreboard feature where you may "live 
>> watch" outstanding requests. You may access to it via the Tomcat Host 
>> Manager Application. As a hint, I may also provide you a short (Unix) script 
>> to readout this from the MBean by HTTP via the jmxproxy-Feature of the 
>> Manager.
>>
>> Greetings
>>
>> Guido
>>
>>> -Original Message-
>>> From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
>>> Sent: Thursday, March 14, 2019 11:10 AM
>>> To: users@tomcat.apache.org
>>> Subject: Crash in http connector random once a day
>>>
>>> The strange thing is, those failed requests with no response are
>>> logged in the tomcat access logs with a 500 http connection error all
>>> at the same time (although they begun with 1-2 min difference) and
>>> after 8 to 24 hours (the URL is alway unique, so I know for sure).
>>
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
Dear Guido,

thanks for the reply. The requests are reaching tomcat, and a thread
is always started, if I look at the current threads on the tomcat
manager I see the following, there are 4 threads that are processing
since 2+ hours:

R ? ? ? ? ? ?
S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET
/manager/status HTTP/1.1
S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=17385 HTTP/1.1
S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=18986 HTTP/1.1
S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=10560 HTTP/1.1
R ? ? ? ? ? ?
S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
/cloudworx/?method=words=10560 HTTP/1.1

I'm not sure what other metrics would be helpful, but your Unix script
wouldn't help much, as I am on a Windows Server 2012 and I would like
to avoid installing Cygwin or something similar.

Regards,
Daniel


Am Do., 14. März 2019 um 12:02 Uhr schrieb Jäkel, Guido :
>
> Dear Daniel,
>
> a request is logged in the access log after it has finished. (In addition, on 
> Tomcat the log is flushed with some delay, but that's not the problem here).
>
> But if the request is stall while processing, there's no hint in the access 
> log. Therefore: Is there an application log that may confirm that the request 
> have reached the application on Tomcat and have start to process?
>
> On Tomcat, there's also a request scoreboard feature where you may "live 
> watch" outstanding requests. You may access to it via the Tomcat Host Manager 
> Application. As a hint, I may also provide you a short (Unix) script to 
> readout this from the MBean by HTTP via the jmxproxy-Feature of the Manager.
>
> Greetings
>
> Guido
>
> >-Original Message-
> >From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >Sent: Thursday, March 14, 2019 11:10 AM
> >To: users@tomcat.apache.org
> >Subject: Crash in http connector random once a day
> >
> >The strange thing is, those failed requests with no response are
> >logged in the tomcat access logs with a 500 http connection error all
> >at the same time (although they begun with 1-2 min difference) and
> >after 8 to 24 hours (the URL is alway unique, so I know for sure).
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Crash in http connector random once a day

2019-03-14 Thread Jäkel , Guido 
Dear Daniel,

a request is logged in the access log after it has finished. (In addition, on 
Tomcat the log is flushed with some delay, but that's not the problem here).

But if the request is stall while processing, there's no hint in the access 
log. Therefore: Is there an application log that may confirm that the request 
have reached the application on Tomcat and have start to process?

On Tomcat, there's also a request scoreboard feature where you may "live watch" 
outstanding requests. You may access to it via the Tomcat Host Manager 
Application. As a hint, I may also provide you a short (Unix) script to readout 
this from the MBean by HTTP via the jmxproxy-Feature of the Manager.

Greetings

Guido

>-Original Message-
>From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
>Sent: Thursday, March 14, 2019 11:10 AM
>To: users@tomcat.apache.org
>Subject: Crash in http connector random once a day
>
>The strange thing is, those failed requests with no response are
>logged in the tomcat access logs with a 500 http connection error all
>at the same time (although they begun with 1-2 min difference) and
>after 8 to 24 hours (the URL is alway unique, so I know for sure).

Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
Hi,

I have long searched and haven't found any real hint how to solve my
problem, so I am adressing you.

Here are my specifications:

Server version:Apache Tomcat/8.5.15
Server built:  May 5 2017 11:03:04 UTC
Server number: 8.5.15.0
OS Name:   Windows Server 2012
OS Version:6.2
Architecture:  amd64
Java Home: C:\Program Files\Java\jre7
JVM Version:   1.7.0_25-b17
JVM Vendor:Oracle Corporation

The tomcat connector I am using has no custom configuration:



I have a PHP Javabridge app (runs PHP CGI to access JAVA functions)
running on tomcat and it's beeing accessed solely through localhost
from other PHP scripts running on IIS. It goes as follows:

1. Request sent to IIS
2. Run PHP on IIS, process the request and delegates a part of the
processing to tomcat
3. Access tomcat through HTTP on localhost:8080 and wait maximum 60s
for the response, retry up to 3 times
4. Run PHP on tomcat, process the request and send data back to the
original PHP process on IIS (on 2.)
5. On IIS read the response from tomcat, process it and send a
response to the original request (on 1.)

For 99% of the time it goes well, we have around 500+ requests beeing
succesfully processed. BUT aprox. once a day there are around 4
requests to tomcat within 3 minutes or so that aren't receiving any
response at all (I have setup a 60s timeout in the IIS PHP process,
usually the tomcat process takes up to 10s to respond).

The strange thing is, those failed requests with no response are
logged in the tomcat access logs with a 500 http connection error all
at the same time (although they begun with 1-2 min difference) and
after 8 to 24 hours (the URL is alway unique, so I know for sure).

In the JAVA logs I can see a "java.net.SocketException: Connection
reset", which is logical, as the PHP process on IIS closed the
connection after the 60s timeout. I tried to set this timeout to
infinite, with the result that the PHP process on IIS waited between 8
and 24 hours, only to get a 500 from tomcat. This time I got in the
JAVA logs a "java.net.SocketException: Connection reset by peer"
(notice the difference "by peer").

I hope I described my problem accurate enough. I'm out of ideas and
I'm grateful for any insight!

Regards,
Daniel

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Crash in http connector random once a day

2019-03-14 Thread Daniel Castilla | thin(k)design 
Hi,

I have long searched and haven't found any real hint how to solve my
problem, so I am adressing you.

Here are my specifications:

Server version:Apache Tomcat/8.5.15
Server built:  May 5 2017 11:03:04 UTC
Server number: 8.5.15.0
OS Name:   Windows Server 2012
OS Version:6.2
Architecture:  amd64
Java Home: C:\Program Files\Java\jre7
JVM Version:   1.7.0_25-b17
JVM Vendor:Oracle Corporation

The tomcat connector I am using has no custom configuration:



I have a PHP Javabridge app (runs PHP CGI to access JAVA functions)
running on tomcat and it's beeing accessed solely through localhost
from other PHP scripts running on IIS. It goes as follows:

1. Request sent to IIS
2. Run PHP on IIS, process the request and delegates a part of the
processing to tomcat
3. Access tomcat through HTTP on localhost:8080 and wait maximum 60s
for the response, retry up to 3 times
4. Run PHP on tomcat, process the request and send data back to the
original PHP process on IIS (on 2.)
5. On IIS read the response from tomcat, process it and send a
response to the original request (on 1.)

For 99% of the time it goes well, we have around 500+ requests beeing
succesfully processed. BUT aprox. once a day there are around 4
requests to tomcat within 3 minutes or so that aren't receiving any
response at all (I have setup a 60s timeout in the IIS PHP process,
usually the tomcat process takes up to 10s to respond).

The strange thing is, those failed requests with no response are
logged in the tomcat access logs with a 500 http connection error all
at the same time (although they begun with 1-2 min difference) and
after 8 to 24 hours (the URL is alway unique, so I know for sure).

In the JAVA logs I can see a "java.net.SocketException: Connection
reset", which is logical, as the PHP process on IIS closed the
connection after the 60s timeout. I tried to set this timeout to
infinite, with the result that the PHP process on IIS waited between 8
and 24 hours, only to get a 500 from tomcat. This time I got in the
JAVA logs a "java.net.SocketException: Connection reset by peer"
(notice the difference "by peer").

I hope I described my problem accurate enough. I'm out of ideas and
I'm grateful for any insight!

Regards,
Daniel

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org