Re: Weirdest Tomcat Behavior Ever?

2020-12-06 Thread Mark Thomas
On 04/12/2020 20:36, Eric Robinson wrote: > We'll begin the packet captures and straces again on Monday. Mark, besides > network and fd tracing, is there anything else you want from strace to make > the analysis more certain? I think the following should be sufficient: - Wireshark trace for

RE: Weirdest Tomcat Behavior Ever?

2020-12-04 Thread Eric Robinson
> -Original Message- > From: Christopher Schultz > Sent: Wednesday, December 2, 2020 10:21 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Mark, > > On 11/26/20 05:14, Mark Thomas wrote: > > On 26/11/2020

Re: Weirdest Tomcat Behavior Ever?

2020-12-02 Thread Christopher Schultz
Mark, On 11/26/20 05:14, Mark Thomas wrote: On 26/11/2020 04:57, Christopher Schultz wrote: After a normal clean-up the parent then calls close on the two file descriptors associated with the pipe for a second time." So the child cleans them up AND the parent cleans them up? Or the parent

Re: Weirdest Tomcat Behavior Ever?

2020-11-26 Thread Mark Thomas
On 26/11/2020 04:57, Christopher Schultz wrote: >> After a normal clean-up the parent then calls close on the two file >> descriptors associated with the pipe for a second time." > > So the child cleans them up AND the parent cleans them up? Or the parent > cleans when up twice? The child

Re: Weirdest Tomcat Behavior Ever?

2020-11-25 Thread Christopher Schultz
Mark, On 11/25/20 22:30, Christopher Schultz wrote: Mark, On 11/25/20 12:49, Mark Thomas wrote: On 25/11/2020 16:46, Christopher Schultz wrote: Strange that the library is getting a direct reference to the response's file descriptor. That seems (to me) to be a tactical mistake in an

Re: Weirdest Tomcat Behavior Ever?

2020-11-25 Thread Christopher Schultz
Mark, On 11/25/20 12:49, Mark Thomas wrote: On 25/11/2020 16:46, Christopher Schultz wrote: Strange that the library is getting a direct reference to the response's file descriptor. That seems (to me) to be a tactical mistake in an implementation. Then again, I'm not looking at what is

Re: Weirdest Tomcat Behavior Ever?

2020-11-25 Thread Mark Thomas
On 25/11/2020 16:46, Christopher Schultz wrote: > Strange that the library is getting a direct reference to the response's > file descriptor. That seems (to me) to be a tactical mistake in an > implementation. Then again, I'm not looking at what is actually being > accomplished, there... it may

Re: [OT] Weirdest Tomcat Behavior Ever?

2020-11-25 Thread Christopher Schultz
Eric, On 11/25/20 09:34, Eric Robinson wrote: -Original Message- From: Christopher Schultz Sent: Tuesday, November 24, 2020 8:11 AM To: users@tomcat.apache.org Subject: Re: Weirdest Tomcat Behavior Ever? Also, v5.0.8 is like 13 years old. Eric, you guys *really* have to upgrade

Re: Weirdest Tomcat Behavior Ever?

2020-11-25 Thread Christopher Schultz
Mark, On 11/24/20 09:56, Mark Thomas wrote: On 24/11/2020 14:11, Christopher Schultz wrote: On 11/20/20 11:08, Mark Thomas wrote: A second look at the strace file from yesterday provided hard evidence of a native library mis-using file descriptors and strong circumstantial evidence

RE: Weirdest Tomcat Behavior Ever?

2020-11-25 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Tuesday, November 24, 2020 8:57 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 24/11/2020 14:11, Christopher Schultz wrote: > > On 11/20/20 11:08, Mark Thomas wrote

RE: Weirdest Tomcat Behavior Ever?

2020-11-25 Thread Eric Robinson
> -Original Message- > From: Christopher Schultz > Sent: Tuesday, November 24, 2020 8:11 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Mark, > > On 11/20/20 11:08, Mark Thomas wrote: > > On 20/11/2020 15:43, Eric Robins

Re: Weirdest Tomcat Behavior Ever?

2020-11-24 Thread Mark Thomas
On 24/11/2020 14:11, Christopher Schultz wrote: > On 11/20/20 11:08, Mark Thomas wrote: >> A second look at the strace file from yesterday provided hard evidence >> of a native library mis-using file descriptors and strong circumstantial >> evidence linking that same library to a specific

Re: Weirdest Tomcat Behavior Ever?

2020-11-24 Thread Christopher Schultz
Mark, On 11/20/20 11:08, Mark Thomas wrote: On 20/11/2020 15:43, Eric Robinson wrote: From: Mark Thomas This information might turn out to be unnecessary. I have been re-looking at the previous logs and I think I have enough evidence to ID a root cause. I'll email off-list as I need to

Re: Weirdest Tomcat Behavior Ever?

2020-11-20 Thread Mark Thomas
On 20/11/2020 15:43, Eric Robinson wrote: >> From: Mark Thomas >> This information might turn out to be unnecessary. I have been re-looking at >> the previous logs and I think I have enough evidence to ID a root cause. I'll >> email off-list as I need to quote logs to prove my point. >> > >

RE: Weirdest Tomcat Behavior Ever?

2020-11-20 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Friday, November 20, 2020 9:32 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 20/11/2020 14:55, Eric Robinson wrote: > >> From: Mark Thomas > >> It looks

Re: Weirdest Tomcat Behavior Ever?

2020-11-20 Thread Mark Thomas
On 20/11/2020 14:55, Eric Robinson wrote: >> From: Mark Thomas >> It looks like you are using MySQL from the data in the strace log. Is that >> correct? What I really need to know is which version of the MySQL JDBC >> driver (Connector/J) are you using? I'm trying to figure out where the root >>

RE: Weirdest Tomcat Behavior Ever?

2020-11-20 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Friday, November 20, 2020 3:17 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 19/11/2020 16:03, Mark Thomas wrote: > > On 19/11/2020 15:55, Eric Robinson wrote: > >

Re: Weirdest Tomcat Behavior Ever?

2020-11-20 Thread Mark Thomas
On 19/11/2020 16:03, Mark Thomas wrote: > On 19/11/2020 15:55, Eric Robinson wrote: >> Unfortunately, the tomcats get restarted every night, so I'll have to watch >> closely for the error to happen today and I will do a thread dump as close >> as possible to the time of the error. > > Sounds

Re: Weirdest Tomcat Behavior Ever?

2020-11-19 Thread Mark Thomas
On 19/11/2020 15:55, Eric Robinson wrote: >> From: Mark Thomas >> Sent: Thursday, November 19, 2020 4:34 AM >> To: users@tomcat.apache.org >> Subject: Re: Weirdest Tomcat Behavior Ever? >> >> On 18/11/2020 16:28, Mark Thomas wrote: >>> On 18/11/2020

RE: Weirdest Tomcat Behavior Ever?

2020-11-19 Thread Eric Robinson
> From: Mark Thomas > Sent: Thursday, November 19, 2020 4:34 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 18/11/2020 16:28, Mark Thomas wrote: > > On 18/11/2020 15:41, Eric Robinson wrote: > > > > >>I tested

Re: Weirdest Tomcat Behavior Ever?

2020-11-19 Thread Mark Thomas
On 18/11/2020 16:28, Mark Thomas wrote: > On 18/11/2020 15:41, Eric Robinson wrote: >>I tested it and we're now capturing file descriptor operations. > I am very curious as to what we are going to see in these logs. Good news. We are a step closer to figuring out what is going on. The new

Re: Weirdest Tomcat Behavior Ever?

2020-11-18 Thread Mark Thomas
On 18/11/2020 15:41, Eric Robinson wrote: >> -Original Message- >> From: Mark Thomas >> Sent: Wednesday, November 18, 2020 3:03 AM >> To: users@tomcat.apache.org >> Subject: Re: Weirdest Tomcat Behavior Ever? >> >> On 13/11/2020 23:46,

RE: Weirdest Tomcat Behavior Ever?

2020-11-18 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Wednesday, November 18, 2020 3:03 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 13/11/2020 23:46, Mark Thomas wrote: > > Eric sent me a copy of the strace (thanks Eric) and

Re: Weirdest Tomcat Behavior Ever?

2020-11-18 Thread Mark Thomas
On 13/11/2020 23:46, Mark Thomas wrote: > Eric sent me a copy of the strace (thanks Eric) and while it is > consistent with what has already been observed, it didn't provide any > new information on the socket / file descriptor being closed. > > I'd like to suggest running again with the

RE: Weirdest Tomcat Behavior Ever?

2020-11-15 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Friday, November 13, 2020 5:47 PM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Eric sent me a copy of the strace (thanks Eric) and while it is consistent > with > wha

Re: Weirdest Tomcat Behavior Ever?

2020-11-13 Thread Mark Thomas
ent: Friday, October 16, 2020 6:11 AM >>> To: Tomcat Users List >>> Subject: Re: Weirdest Tomcat Behavior Ever? >>> >>> Hi Eric, >>> >>> These weird situations are sometimes best looked at by confirming what >> the >>> OS is seein

Re: Weirdest Tomcat Behavior Ever?

2020-11-13 Thread Paul Carter-Brown
-Original Message- > > From: Paul Carter-Brown > > Sent: Friday, October 16, 2020 6:11 AM > > To: Tomcat Users List > > Subject: Re: Weirdest Tomcat Behavior Ever? > > > > Hi Eric, > > > > These weird situations are sometimes best looked at

RE: Weirdest Tomcat Behavior Ever?

2020-11-13 Thread Eric Robinson
> From: Thomas Meyer > Sent: Friday, November 13, 2020 9:37 AM > To: Tomcat Users List ; Mark Thomas > ; users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > > > Am 13. November 2020 10:06:18 MEZ schrieb Mark Thomas > : > >On 12/11/2020 14

Re: Weirdest Tomcat Behavior Ever?

2020-11-13 Thread Thomas Meyer
Am 13. November 2020 10:06:18 MEZ schrieb Mark Thomas : >On 12/11/2020 14:19, Eric Robinson wrote: >>> From: Mark Thomas > > > >>> I keep coming back to this. Something triggered this problem (note >that >>> trigger not necessarily the same as root cause). Given that the app, >Tomcat >>> and

RE: Weirdest Tomcat Behavior Ever?

2020-11-13 Thread Eric Robinson
> -Original Message- > From: Paul Carter-Brown > Sent: Friday, October 16, 2020 6:11 AM > To: Tomcat Users List > Subject: Re: Weirdest Tomcat Behavior Ever? > > Hi Eric, > > These weird situations are sometimes best looked at by confirming what the >

RE: Weirdest Tomcat Behavior Ever?

2020-11-13 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Friday, November 13, 2020 3:06 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 12/11/2020 14:19, Eric Robinson wrote: > >> From: Mark Thomas > > > > >>

Re: Weirdest Tomcat Behavior Ever?

2020-11-13 Thread Mark Thomas
On 12/11/2020 14:19, Eric Robinson wrote: >> From: Mark Thomas >> I keep coming back to this. Something triggered this problem (note that >> trigger not necessarily the same as root cause). Given that the app, Tomcat >> and JVM versions didn't change that again points to some other component.

RE: Weirdest Tomcat Behavior Ever?

2020-11-12 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Thursday, November 12, 2020 4:08 AM > To: Tomcat Users List ; Eric Robinson > > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 11/11/2020 22:48, Eric Robinson wrote: > >> -Original Message-

Re: Weirdest Tomcat Behavior Ever?

2020-11-12 Thread Mark Thomas
On 11/11/2020 22:48, Eric Robinson wrote: >> -Original Message- >> From: Mark Thomas >> Sent: Monday, November 9, 2020 5:59 AM >> To: users@tomcat.apache.org >> Subject: Re: Weirdest Tomcat Behavior Ever? >> >> Eric, >> >> Time to

RE: Weirdest Tomcat Behavior Ever?

2020-11-11 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Monday, November 9, 2020 5:59 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Eric, > > Time to prune the history and provide another summary I think. This > summary

Re: Weirdest Tomcat Behavior Ever?

2020-11-09 Thread Mark Thomas
Eric, Time to prune the history and provide another summary I think. This summary isn't complete. There is more information in the history of the thread. I'm trying to focus on what seems to be the key information. Overview: A small number of requests are receiving a completely empty (no

RE: Weirdest Tomcat Behavior Ever?

2020-11-06 Thread Eric Robinson
> > -Original Message- > > From: Stefan Mayr > > Sent: Thursday, November 5, 2020 4:24 PM > > To: users@tomcat.apache.org > > Subject: Re: Weirdest Tomcat Behavior Ever? > > > > Am 03.11.2020 um 16:05 schrieb Eric Robinson: > > >>

RE: Weirdest Tomcat Behavior Ever?

2020-11-06 Thread Eric Robinson
> -Original Message- > From: Stefan Mayr > Sent: Thursday, November 5, 2020 4:24 PM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Am 03.11.2020 um 16:05 schrieb Eric Robinson: > >> -Original Message- > >>

Re: Weirdest Tomcat Behavior Ever?

2020-11-05 Thread Stefan Mayr
Am 03.11.2020 um 16:05 schrieb Eric Robinson: >> -Original Message- >> From: Eric Robinson >> Sent: Tuesday, November 3, 2020 8:21 AM >> To: Tomcat Users List >> Subject: RE: Weirdest Tomcat Behavior Ever? >> >>> From: Mark Thomas

RE: Weirdest Tomcat Behavior Ever?

2020-11-04 Thread Eric Robinson
> From: Mark Thomas > Sent: Wednesday, November 4, 2020 11:39 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 03/11/2020 15:05, Eric Robinson wrote: > >> From: Eric Robinson > >>> From: Mark Thomas > >

Re: Weirdest Tomcat Behavior Ever?

2020-11-04 Thread Mark Thomas
On 03/11/2020 15:05, Eric Robinson wrote: >> From: Eric Robinson >>> From: Mark Thomas >>> Progress. The first attempt to write to the socket triggers the >>> following >>> exception: >>> >>> 02-Nov-2020 14:33:54.083 FINE [http-bio-3016-exec-13] >>>

RE: Weirdest Tomcat Behavior Ever?

2020-11-03 Thread Eric Robinson
> From: Christopher Schultz > Sent: Tuesday, November 3, 2020 9:26 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Eric, > > On 11/3/20 10:05, Eric Robinson wrote: > >> -Original Message- > >> From: Eric Robins

Re: Weirdest Tomcat Behavior Ever?

2020-11-03 Thread Christopher Schultz
Eric, On 11/3/20 10:05, Eric Robinson wrote: -Original Message- From: Eric Robinson Sent: Tuesday, November 3, 2020 8:21 AM To: Tomcat Users List Subject: RE: Weirdest Tomcat Behavior Ever? From: Mark Thomas Sent: Tuesday, November 3, 2020 2:06 AM To: Tomcat Users List Subject: Re

Re: [OT] Weirdest Tomcat Behavior Ever?

2020-11-03 Thread Christopher Schultz
Mark, On 11/3/20 03:05, Mark Thomas wrote: On 02/11/2020 12:16, Eric Robinson wrote: Gotcha, thanks for the clarification. Let's see what happens when the users start hitting it at 8:00 am Eastern. Progress. The first attempt to write to the socket triggers the following exception:

RE: Weirdest Tomcat Behavior Ever?

2020-11-03 Thread Eric Robinson
> -Original Message- > From: Eric Robinson > Sent: Tuesday, November 3, 2020 8:21 AM > To: Tomcat Users List > Subject: RE: Weirdest Tomcat Behavior Ever? > > > From: Mark Thomas > > Sent: Tuesday, November 3, 2020 2:06 AM > > To: Tomcat Users Li

RE: Weirdest Tomcat Behavior Ever?

2020-11-03 Thread Eric Robinson
> From: Mark Thomas > Sent: Tuesday, November 3, 2020 2:06 AM > To: Tomcat Users List > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 02/11/2020 12:16, Eric Robinson wrote: > > > > > Gotcha, thanks for the clarification. Let's see what happens when the us

Re: Weirdest Tomcat Behavior Ever?

2020-11-03 Thread Mark Thomas
On 02/11/2020 12:16, Eric Robinson wrote: > Gotcha, thanks for the clarification. Let's see what happens when the users > start hitting it at 8:00 am Eastern. Progress. The first attempt to write to the socket triggers the following exception: 02-Nov-2020 14:33:54.083 FINE

RE: Weirdest Tomcat Behavior Ever?

2020-11-02 Thread Eric Robinson
> From: Mark Thomas > Sent: Monday, November 2, 2020 5:38 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 02/11/2020 11:18, Eric Robinson wrote: > >> -Original Message- > >> From: Mark Thomas > >

Re: Weirdest Tomcat Behavior Ever?

2020-11-02 Thread Mark Thomas
On 02/11/2020 11:18, Eric Robinson wrote: >> -Original Message- >> From: Mark Thomas >> Sent: Sunday, November 1, 2020 11:50 AM >> To: users@tomcat.apache.org >> Subject: Re: Weirdest Tomcat Behavior Ever? >> >> On 01/11/2020 16:25, Mark Thoma

RE: Weirdest Tomcat Behavior Ever?

2020-11-02 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Sunday, November 1, 2020 11:50 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 01/11/2020 16:25, Mark Thomas wrote: > > > > > > Keeping the previous log

Re: Weirdest Tomcat Behavior Ever?

2020-11-01 Thread Mark Thomas
On 01/11/2020 16:25, Mark Thomas wrote: > > > Keeping the previous logs for reference: > >>> Source Time Activity >>> >>> pcap15:14:25.375451 SYN proxy to Tomcat >>> pcap15:14:25.375493 SYN, ACK Tomcat to proxy >>> pcap

Re: Weirdest Tomcat Behavior Ever?

2020-11-01 Thread Mark Thomas
Keeping the previous logs for reference: >> Source Time Activity >> >> pcap15:14:25.375451 SYN proxy to Tomcat >> pcap15:14:25.375493 SYN, ACK Tomcat to proxy >> pcap15:14:25.375839 ACK proxy to Tomcat >> pcap

RE: Weirdest Tomcat Behavior Ever?

2020-10-29 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Thursday, October 29, 2020 5:45 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 28/10/2020 20:32, Mark Thomas wrote: > > > > > I have the off-list mail and wi

Re: Weirdest Tomcat Behavior Ever?

2020-10-29 Thread Mark Thomas
On 29/10/2020 10:44, Mark Thomas wrote: > I think more debug logging is required. I am currently working on that. v2 of the debug logging patch is: http://home.apache.org/~markt/dev/v7.0.72-custom-patch-v2/ Full disclosure: This patch is more invasive. I had to wrap the java.net.Socket

Re: Weirdest Tomcat Behavior Ever?

2020-10-29 Thread Mark Thomas
On 28/10/2020 20:32, Mark Thomas wrote: > I have the off-list mail and will start looking at the logs shortly. Progress. I think. I'll start with the following summary of the log data. Source Time Activity pcap

Re: Weirdest Tomcat Behavior Ever?

2020-10-28 Thread Mark Thomas
On 28/10/2020 20:06, Eric Robinson wrote: >> From: Eric Robinson >> Sent: Tuesday, October 27, 2020 11:33 PM >> To: Tomcat Users List >> Subject: RE: Weirdest Tomcat Behavior Ever? >> >>> From: Mark Thomas >>> Sent: Tuesday, October 27, 2020 12:0

RE: Weirdest Tomcat Behavior Ever?

2020-10-28 Thread Eric Robinson
> From: Eric Robinson > Sent: Tuesday, October 27, 2020 11:33 PM > To: Tomcat Users List > Subject: RE: Weirdest Tomcat Behavior Ever? > > > From: Mark Thomas > > Sent: Tuesday, October 27, 2020 12:06 PM > > To: users@tomcat.apache.org > > Sub

RE: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Eric Robinson
> From: Mark Thomas > Sent: Tuesday, October 27, 2020 12:06 PM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 27/10/2020 16:29, Eric Robinson wrote: > >> On 27/10/2020 15:22, Eric Robinson wrote: > > > > >>&g

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas
On 27/10/2020 16:29, Eric Robinson wrote: >> On 27/10/2020 15:22, Eric Robinson wrote: >>> I had switched to the NIO connector last week. Is that why the logs are >>> still >> at 0 bytes? >> >> Yes. I only added the debug logging to BIO. >> >> Somewhere in a previous thread I recommended

RE: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Eric Robinson
> On 27/10/2020 15:22, Eric Robinson wrote: > >> On 27/10/2020 09:16, Mark Thomas wrote: > >>> On 27/10/2020 04:43, Eric Robinson wrote: > >>> > >>> > >>> > Any changes in the Nginx configuration in the relevant timescale? > > >> > >> The last change to the nginx config

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas
On 27/10/2020 15:22, Eric Robinson wrote: >> On 27/10/2020 09:16, Mark Thomas wrote: >>> On 27/10/2020 04:43, Eric Robinson wrote: >>> >>> >>> Any changes in the Nginx configuration in the relevant timescale? >> >> The last change to the nginx config files was on 8/21.

RE: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Eric Robinson
> On 27/10/2020 09:16, Mark Thomas wrote: > > On 27/10/2020 04:43, Eric Robinson wrote: > > > > > > > >> Any changes in the Nginx configuration in the relevant timescale? > >> > > The last change to the nginx config files was on 8/21. The first > report of problems from the

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas
On 27/10/2020 09:16, Mark Thomas wrote: > On 27/10/2020 04:43, Eric Robinson wrote: > > > >> Any changes in the Nginx configuration in the relevant timescale? >> The last change to the nginx config files was on 8/21. The first report of problems from the users in question

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas
On 27/10/2020 04:43, Eric Robinson wrote: > Any changes in the Nginx configuration in the relevant timescale? > >>> >>> The last change to the nginx config files was on 8/21. The first >>> report of problems from the users in question was on 9/16. There is >>> another set of users on a

RE: Weirdest Tomcat Behavior Ever?

2020-10-26 Thread Eric Robinson
> -Original Message- > From: Eric Robinson > Sent: Monday, October 26, 2020 11:37 PM > To: Tomcat Users List > Subject: RE: Weirdest Tomcat Behavior Ever? > > > > On 26/10/2020 10:26, Mark Thomas wrote: > > > > O

RE: Weirdest Tomcat Behavior Ever?

2020-10-26 Thread Eric Robinson
> > On 26/10/2020 10:26, Mark Thomas wrote: > > > On 24/10/2020 01:32, Eric Robinson wrote: > > > > > > > > > > > -Original Message- > > From: Mark Thomas > > > > > > > > > > > The failed request: > > - Completes in ~6ms > > >>> > > >>> I think we've seen the failed

Re: Weirdest Tomcat Behavior Ever?

2020-10-26 Thread Konstantin Kolinko
вт, 27 окт. 2020 г. в 00:07, Eric Robinson : > > > On 26/10/2020 10:26, Mark Thomas wrote: > > > On 24/10/2020 01:32, Eric Robinson wrote: > > > > > > At this point I'd echo Konstantin's recommendation to add the > > > following system property: > > >

RE: Weirdest Tomcat Behavior Ever?

2020-10-26 Thread Eric Robinson
> On 26/10/2020 10:26, Mark Thomas wrote: > > On 24/10/2020 01:32, Eric Robinson wrote: > > > > > > > -Original Message- > From: Mark Thomas > > > > > > > The failed request: > - Completes in ~6ms > >>> > >>> I think we've seen the failed requests take as much as

Re: Weirdest Tomcat Behavior Ever?

2020-10-26 Thread Mark Thomas
On 26/10/2020 10:26, Mark Thomas wrote: > On 24/10/2020 01:32, Eric Robinson wrote: > > > -Original Message- From: Mark Thomas > > > The failed request: - Completes in ~6ms >>> >>> I think we've seen the failed requests take as much as 50ms. > > Ack. That is

Re: Weirdest Tomcat Behavior Ever?

2020-10-26 Thread Mark Thomas
On 24/10/2020 01:32, Eric Robinson wrote: >>> -Original Message- >>> From: Mark Thomas >>> The failed request: >>> - Completes in ~6ms >> >> I think we've seen the failed requests take as much as 50ms. Ack. That is still orders of magnitude smaller that the timeout and consistent

Re: Weirdest Tomcat Behavior Ever?

2020-10-24 Thread Konstantin Kolinko
сб, 24 окт. 2020 г. в 03:08, Eric Robinson : > > > - Answers to the additional questions would be nice but the access log > > %b value for a failed request is the key piece of information required > > at this point. > > > > Good news! I enabled that parameter a few days ago and we have already

RE: Weirdest Tomcat Behavior Ever?

2020-10-23 Thread Eric Robinson
> -Original Message- > From: Eric Robinson > Sent: Friday, October 23, 2020 7:09 PM > To: Tomcat Users List > Subject: RE: Weirdest Tomcat Behavior Ever? > > Hi Mark -- > > Thanks tons for digging into this. See my answers below. > > > -Origina

RE: Weirdest Tomcat Behavior Ever?

2020-10-23 Thread Eric Robinson
Hi Mark -- Thanks tons for digging into this. See my answers below. > -Original Message- > From: Mark Thomas > Sent: Friday, October 23, 2020 5:09 PM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Hi Eric (and those following alo

Re: Weirdest Tomcat Behavior Ever?

2020-10-23 Thread Mark Thomas
Hi Eric (and those following along), Eric sent me some network captures off-list from multiple points in the network from the system where this is happening. The following is the summary so far: Overview: A small number of requests are receiving a completely empty (no headers, no body) response.

RE: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Friday, October 16, 2020 8:02 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 16/10/2020 12:37, Eric Robinson wrote: > >> From: Mark Thomas > > > > >> I

Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Mark Thomas
On 16/10/2020 12:37, Eric Robinson wrote: >> From: Mark Thomas >> I'd like to see those screen shots please. Better still would be access to >> the >> captures themselves (just the relevant connections not the whole thing). I >> believe what you are telling us but long experience tells me it

Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Daniel Skiles
ark -- > > Those are great questions. See answers below. > > > > -Original Message- > > From: Mark Thomas > > Sent: Friday, October 16, 2020 2:20 AM > > To: users@tomcat.apache.org > > Subject: Re: Weirdest Tomcat Behavior Ever? >

RE: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Eric Robinson
> -Original Message- > From: Mark Thomas > Sent: Friday, October 16, 2020 5:17 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 16/10/2020 10:05, Eric Robinson wrote: > > Hi Mark -- > > > > Those

Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Paul Carter-Brown
-Original Message- > >> From: Mark Thomas > >> Sent: Friday, October 16, 2020 2:20 AM > >> To: users@tomcat.apache.org > >> Subject: Re: Weirdest Tomcat Behavior Ever? > >> > >> On 16/10/2020 00:27, Eric Robinson wrote: > >&

Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Mark Thomas
On 16/10/2020 10:05, Eric Robinson wrote: > Hi Mark -- > > Those are great questions. See answers below. > > >> -Original Message- >> From: Mark Thomas >> Sent: Friday, October 16, 2020 2:20 AM >> To: users@tomcat.apache.org >> Subject: Re:

RE: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Eric Robinson
> > 6. What timeouts are configured for the Connector? > > > > Sorry, which connector are you referring to? > Sorry, I'm a dummy. Obviously you mean the tomcat connector. connectionTimeout="2" -Eric Disclaimer : This email and any files transmitted with it are confidential and intended

RE: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Eric Robinson
Hi Mark -- Those are great questions. See answers below. > -Original Message- > From: Mark Thomas > Sent: Friday, October 16, 2020 2:20 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 16/10/2020 00:27

Re: Weirdest Tomcat Behavior Ever?

2020-10-16 Thread Mark Thomas
On 16/10/2020 00:27, Eric Robinson wrote: > The localhost_access log shows a request received and an HTTP 200 response > sent, as follows... > > 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > /app/code.jsp?gizmoid=64438=5=2020-10-15=0=0=71340=321072=40696=0.0715816=15102020125245.789063

Weirdest Tomcat Behavior Ever?

2020-10-15 Thread Eric Robinson
Has anyone ever seen a situation where tomcat occasionally fails to send responses but still logs them? On a CentOS 7.5 server running in Azure with tomcat 7.0.76 with java 1.0.0_221, everything runs fine 99.99% of the time, but that last hundredth of a percent is a bitch. However,