RE: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Eric Robinson 
> From: Mark Thomas 
> Sent: Tuesday, October 27, 2020 12:06 PM
> To: users@tomcat.apache.org
> Subject: Re: Weirdest Tomcat Behavior Ever?
>
> On 27/10/2020 16:29, Eric Robinson wrote:
> >> On 27/10/2020 15:22, Eric Robinson wrote:
>
> 
>
> >>> I had switched to the NIO connector last week. Is that why the logs
> >>> are still
> >> at 0 bytes?
> >>
> >> Yes. I only added the debug logging to BIO.
> >>
> >> Somewhere in a previous thread I recommended switching back to BIO as
> >> the code is simpler and therefore easier to debug.
> >>
> >> Note that my previous analysis of the access log valve entries was
> >> based on the assumption you had switched back to BIO.
> >>
> >> Given everything else is in place, I'd suggest switching back to BIO
> >> when you can, waiting for the issue to re-appear and then looking at the
> debug logs.
> >>
> >> Mark
> >>
> >>
> >
> > Oh man, I must have missed the recommendation to switch back to BIO. I
> will do that ASAP, most likely this evening. My apologies for causing the
> wasted effort.
>
> No worries.
>
> Looking at the NIO code, if you had switched back to BIO last night I'm fairly
> sure we'd now be adding the debug logging to BIO anyway so we'll end up in
> the same place.
>
> Now seems like a good time to point out that, as email based debugging
> sessions go, this is very much at the better end. Having someone as
> responsive as you are, who actually answers the questions asked and is
> willing to run custom patches to collect debug info makes it much more likely
> that not only will we reach a successful conclusion, but that we'll get there
> reasonably quickly.
>

All I can say is that I greatly appreciate the detailed help!

Tomcat has been put back into BIO mode and has been restarted. I confirmed that 
the debug0.log file is now accumulating data.

I am very eager to see what turns up in the logs Wednesday morning.


> Thanks,
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

Our webapp is running very slowly on one particular customer box

2020-10-27 Thread James H. H. Lampert 

This is related to my query (thanks, Mr. Gregg) about "Tenured SOA."

It seems that on one of our customer installations, our webapp gets into 
a state of running very slowly, and the dedicated subsystem it's running 
in is showing massive levels of page-faulting.


I've compared the GC stats of the "problem" system with one that's 
actually got more users connected, but isn't experiencing performance 
issues. It seems that they're both going to GC about every 30-50 
seconds, but GC on the "problem" system appears to be somewhat less 
effective.


Also, I've looked at the threads on both. On the system that is behaving 
normally, the "GC Slave" threads (7 of them) are showing total CPU (at 
this moment) of around 150 seconds each, and Aux I/O of mostly zero, 
with one showing 1 and one showing 3. Conversely, on the "problem" 
system, I'm seeing 15(!) GC Slave threads, each with total CPU under 6 
seconds each, but Aux I/O ranging from 5800 to over 8000.


I'm not sure what to make of this. In both cases, Tomcat's JVM is 
running in a subsystem of its own, with a private memory pool of around 
7G, and they're both running with -Xms4096m -Xmx5120m.


--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas 
On 27/10/2020 16:29, Eric Robinson wrote:
>> On 27/10/2020 15:22, Eric Robinson wrote:



>>> I had switched to the NIO connector last week. Is that why the logs are 
>>> still
>> at 0 bytes?
>>
>> Yes. I only added the debug logging to BIO.
>>
>> Somewhere in a previous thread I recommended switching back to BIO as
>> the code is simpler and therefore easier to debug.
>>
>> Note that my previous analysis of the access log valve entries was based on
>> the assumption you had switched back to BIO.
>>
>> Given everything else is in place, I'd suggest switching back to BIO when you
>> can, waiting for the issue to re-appear and then looking at the debug logs.
>>
>> Mark
>>
>>
> 
> Oh man, I must have missed the recommendation to switch back to BIO. I will 
> do that ASAP, most likely this evening. My apologies for causing the wasted 
> effort.

No worries.

Looking at the NIO code, if you had switched back to BIO last night I'm
fairly sure we'd now be adding the debug logging to BIO anyway so we'll
end up in the same place.

Now seems like a good time to point out that, as email based debugging
sessions go, this is very much at the better end. Having someone as
responsive as you are, who actually answers the questions asked and is
willing to run custom patches to collect debug info makes it much more
likely that not only will we reach a successful conclusion, but that
we'll get there reasonably quickly.

Thanks,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk "Can not determine the proper size for pid_t" on macOS 10.15.7

2020-10-27 Thread Christopher Schultz 

Brian

On 10/26/20 15:33, Paquin, Brian wrote:

I’m trying to build httpd and mod_jk for the first time on a macOS 10.15.7 box. 
XCode 12.1 is installed and I was able to compile OpenSSL 1.1.1g.
I got an error “Can not determine the proper size for pid_t” when compiling 
httpd (v2.4.46) with included apr (v1.7.0).
This issue https://bz.apache.org/bugzilla/show_bug.cgi?id=64753 provided a diff patch 
that adds “#include ” in a number of locations.
Applying this patch allowed me to compile httpd!


Weird. pid_it is defined in  and yet the patch adds 
 to fix it.


I don't have access to my Catalina machine right now, but my clang-based 
Mojave machine still says to use  when you "man getpid":


"
GETPID(2)   BSD System Calls Manual

NAME
 getpid, getppid -- get parent or calling process identification

SYNOPSIS
 #include 

 pid_t
 getpid(void);
"

$ cc --version
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin


Now I am trying to compile mod_jk (v1.2.48), and I get the same error.
Does someone have a patch file I can use to get around this issue?

$ ./configure CFLAGS='-arch x86_64' APXSLDFLAGS='-arch x86_64' 
--with-apxs=/usr/local/apache2/bin/apxs

$ make

Making all in common
/usr/local/apache-2.4.46/build/libtool --silent --mode=compile gcc -I. 
-I/usr/local/apache-2.4.46/include -arch x86_64 -DHAVE_CONFIG_H -arch x86_64  
-DHAVE_APR  -I/usr/local/apache-2.4.46/include 
-I/usr/local/apache-2.4.46/include -arch x86_64 -DHAVE_CONFIG_H -DDARWIN 
-DSIGPROCMASK_SETS_THREAD_MASK -DDARWIN_10 -c jk_ajp12_worker.c -o 
jk_ajp12_worker.lo
In file included from jk_ajp12_worker.c:25:
In file included from ./jk_ajp12_worker.h:26:
In file included from ./jk_logger.h:26:
In file included from ./jk_global.h:340:
./jk_types.h:56:2: error: Can not determine the proper size for pid_t
#error Can not determine the proper size for pid_t
  ^
./jk_types.h:62:2: error: Can not determine the proper size for pthread_t
#error Can not determine the proper size for pthread_t
  ^
2 errors generated.
make[1]: *** [jk_ajp12_worker.lo] Error 1
make: *** [all-recursive] Error 1
$


I'm sorry, I have no idea how configure does its magic. The 
auto-generated jk_types.h looks like a hand-wavy template to me.


You can probably hack it briefly by running "configure" (which you 
already did) and then hand-editing include/jk_types.h (ignoring the 
warning NOT to hand-edit it!) and manually adding:


#include 

to the top.

Give that a try and see if it works.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Eric Robinson 
> On 27/10/2020 15:22, Eric Robinson wrote:
> >> On 27/10/2020 09:16, Mark Thomas wrote:
> >>> On 27/10/2020 04:43, Eric Robinson wrote:
> >>>
> >>> 
> >>>
>  Any changes in the Nginx configuration in the relevant timescale?
> 
> >>
> >> The last change to the nginx config files was on 8/21. The first
> >> report of problems from the users in question was on 9/16. There
> >> is another set of users on a different tomcat instance who
> >> reported issues around 8/26, 5 days after nginx config change. It
> >> seems unlikely to be related. Also, I can't imagine what nginx
> >> could be sending that would induce the upstream tomcat to behave
> this way.
> >>>
> >>> If there is some sort of retaining references to request/response/etc.
> >>> at the root of this then that sort of issue is very sensitive to timing.
> >>> For example, I have had reliable reproduction cases in the past that
> >>> stopped working with the addition of a single debug statement. Any
> >>> sort of configuration change might have changed the timing
> >>> sufficiently to trigger the issue
> >>>
> >>> At this point, I'd say the Nginx config change might be a potential
> >>> trigger if the root cause is retaining references.
> >>>
>  Any updates to the application in the relevant timescale?
> 
> >>
> >> Their application was patched to a newer version on 6/5.
> >>>
> >>> That seems far enough away to be unlikely.
> >>>
>  Any features users started using that hadn't been used before
>  in that timescale?
> >>
> >> That one I couldn't answer, as we are only the hosting facility
> >> and we are not in the loop when it comes to the users' workflow,
> >> but it seems unlikely given the nature of their business.
> >>>
> >>> Fair enough. That one was a bit of a shot in the dark.
> >>>
> >>> 
> >>>
> >> 1. Now that you have provided this patch, should I still enable
> >> RECYCLE_FACADES=true?
> >>>
> >>> I'd recommend yes. At least until the issue is resolved.
> >>>
> >> 2. The servers in question are multi-tenanted. There are 17
> >> instances of tomcat, each running on a different set of ports and
> >> out of a separate directory tree, but they are all started with
> >> essentially the same init script, which exports certain
> >> site-specific path variables and executes tomcat from the
> >> specified folder structure. Can you think of any potential issues
> >> where making this change for one instance could have a negative
> >> effect on any of the other instances? Probably not, but just
> >> being careful. I will have these changes implemented during our
> >> nightly maintenance window will begin to gather relevant logs
> > first thing tomorrow!
> >>>
> >>> I can't think of any side effects.
> >>>
> >>
> >> --Eric
> >
> > Mark, the changes have been made per your instructions and tomcat
> > has been restarted. The debug0.log, and debug0.log.lck files were
> > created in the directory, but they currently both have 0 bytes.
> >>>
> >>> Drat. That suggests something isn't quite right as the logs should
> >>> start filling up as soon as a single request is made. I'll double
> >>> check my instructions if you could double check your end.
> >>
> >> I've clarified a few things in the instructions and confirmed they
> >> work with my local 7.0.72 build.
> >>
> >> Note: You will need to be using the BIO connector
> >>
> >
> > I had switched to the NIO connector last week. Is that why the logs are 
> > still
> at 0 bytes?
>
> Yes. I only added the debug logging to BIO.
>
> Somewhere in a previous thread I recommended switching back to BIO as
> the code is simpler and therefore easier to debug.
>
> Note that my previous analysis of the access log valve entries was based on
> the assumption you had switched back to BIO.
>
> Given everything else is in place, I'd suggest switching back to BIO when you
> can, waiting for the issue to re-appear and then looking at the debug logs.
>
> Mark
>
>

Oh man, I must have missed the recommendation to switch back to BIO. I will do 
that ASAP, most likely this evening. My apologies for causing the wasted effort.

> >
> >
> >> Mark
> >>
> >>>
> >>> Konstantin noted there was no source provided. I've pushed the
> >>> branch to
> >>> https://github.com/markt-asf/tomcat/tree/debug-7.0.72 so you can see
> >>> the changes I made.
> >>>
>  Also, RECYCLE_FACADES has been enabled and I confirmed that it is
> >> referenced in the logs as follows...
> 
>  INFO: Command line argument:
>  -Dorg.apache.catalina.connector.RECYCLE_FACADES=true
> >>>
> >>> Great.
> >>>
> >>> Mark
> >>>
> >>> 
> >>> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>
> >>
> >>

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas 
On 27/10/2020 15:22, Eric Robinson wrote:
>> On 27/10/2020 09:16, Mark Thomas wrote:
>>> On 27/10/2020 04:43, Eric Robinson wrote:
>>>
>>> 
>>>
 Any changes in the Nginx configuration in the relevant timescale?

>>
>> The last change to the nginx config files was on 8/21. The first
>> report of problems from the users in question was on 9/16. There is
>> another set of users on a different tomcat instance who reported
>> issues around 8/26, 5 days after nginx config change. It seems
>> unlikely to be related. Also, I can't imagine what nginx could be
>> sending that would induce the upstream tomcat to behave this way.
>>>
>>> If there is some sort of retaining references to request/response/etc.
>>> at the root of this then that sort of issue is very sensitive to timing.
>>> For example, I have had reliable reproduction cases in the past that
>>> stopped working with the addition of a single debug statement. Any
>>> sort of configuration change might have changed the timing
>>> sufficiently to trigger the issue
>>>
>>> At this point, I'd say the Nginx config change might be a potential
>>> trigger if the root cause is retaining references.
>>>
 Any updates to the application in the relevant timescale?

>>
>> Their application was patched to a newer version on 6/5.
>>>
>>> That seems far enough away to be unlikely.
>>>
 Any features users started using that hadn't been used before in
 that timescale?
>>
>> That one I couldn't answer, as we are only the hosting facility and
>> we are not in the loop when it comes to the users' workflow, but it
>> seems unlikely given the nature of their business.
>>>
>>> Fair enough. That one was a bit of a shot in the dark.
>>>
>>> 
>>>
>> 1. Now that you have provided this patch, should I still enable
>> RECYCLE_FACADES=true?
>>>
>>> I'd recommend yes. At least until the issue is resolved.
>>>
>> 2. The servers in question are multi-tenanted. There are 17
>> instances of tomcat, each running on a different set of ports and
>> out of a separate directory tree, but they are all started with
>> essentially the same init script, which exports certain
>> site-specific path variables and executes tomcat from the specified
>> folder structure. Can you think of any potential issues where
>> making this change for one instance could have a negative effect on
>> any of the other instances? Probably not, but just being careful. I
>> will have these changes implemented during our nightly maintenance
>> window will begin to gather relevant logs
> first thing tomorrow!
>>>
>>> I can't think of any side effects.
>>>
>>
>> --Eric
>
> Mark, the changes have been made per your instructions and tomcat
> has been restarted. The debug0.log, and debug0.log.lck files were
> created in the directory, but they currently both have 0 bytes.
>>>
>>> Drat. That suggests something isn't quite right as the logs should
>>> start filling up as soon as a single request is made. I'll double
>>> check my instructions if you could double check your end.
>>
>> I've clarified a few things in the instructions and confirmed they work with
>> my local 7.0.72 build.
>>
>> Note: You will need to be using the BIO connector
>>
> 
> I had switched to the NIO connector last week. Is that why the logs are still 
> at 0 bytes?

Yes. I only added the debug logging to BIO.

Somewhere in a previous thread I recommended switching back to BIO as
the code is simpler and therefore easier to debug.

Note that my previous analysis of the access log valve entries was based
on the assumption you had switched back to BIO.

Given everything else is in place, I'd suggest switching back to BIO
when you can, waiting for the issue to re-appear and then looking at the
debug logs.

Mark


> 
> 
>> Mark
>>
>>>
>>> Konstantin noted there was no source provided. I've pushed the branch
>>> to
>>> https://github.com/markt-asf/tomcat/tree/debug-7.0.72 so you can see
>>> the changes I made.
>>>
 Also, RECYCLE_FACADES has been enabled and I confirmed that it is
>> referenced in the logs as follows...

 INFO: Command line argument:
 -Dorg.apache.catalina.connector.RECYCLE_FACADES=true
>>>
>>> Great.
>>>
>>> Mark
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
> 
> Disclaimer : This email and any files transmitted with it are confidential 
> and intended solely for intended recipients. If you are not the named 
> addressee you should not disseminate, distribute, copy or alter this email. 
> Any

RE: Default Application questions

2020-10-27 Thread jonmcalexander 
Thank you! This answered my question.


Dream * Excel * Explore * Inspire
Jon McAlexander
Infrastructure Engineer
Asst Vice President

Middleware Product Engineering
Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com


This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.


-Original Message-
From: Mark Thomas  
Sent: Tuesday, October 27, 2020 4:49 AM
To: users@tomcat.apache.org
Subject: Re: Default Application questions

On 26/10/2020 19:37, jonmcalexan...@wellsfargo.com.INVALID wrote:
> I'm doing some documentation cleanup. When was the balancer app, if it 
> existed, removed from the ootb tomcat applications?

It was in 5.0.x (5.0.15 onwards) and 5.5.x.
http://tomcat.apache.org/tomcat-5.5-doc/balancer-howto.html

> Was there ever a webdav app, or just the WebdavServlet class?

Yes there was a webdav app. It was present in 4.0.x, 4.1.x, 5.0.x, and 5.5.x.

I couldn't find a reference in the change log for their removal.

Mark


> 
> Thanks,
> 
> 
> Dream * Excel * Explore * Inspire
> Jon McAlexander
> Infrastructure Engineer
> Asst Vice President
> 
> Middleware Product Engineering
> Enterprise CIO | Platform Services | Middleware | Infrastructure 
> Solutions
> 
> 8080 Cobblestone Rd | Urbandale, IA 50322
> MAC: F4469-010
> Tel 515-988-2508 | Cell 515-988-2508
> 
> jonmcalexan...@wellsfargo.com
> 
> 
> This message may contain confidential and/or privileged information. If you 
> are not the addressee or authorized to receive this for the addressee, you 
> must not use, copy, disclose, or take any action based on this message or any 
> information herein. If you have received this message in error, please advise 
> the sender immediately by reply e-mail and delete this message. Thank you for 
> your cooperation.
> 
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: FW: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

2020-10-27 Thread Arshiya Shariff 
Hi all, 

Thanks for the update Christopher.

I tried modifying our application as below i.e by adding a read Listener and 
processing the requests after all the data is read (onDataAvailable()) . 
But I can still see the StackOverFlow Error printed .

AsyncContext asyncContext = req.startAsync( req, resp );
ServletInputStream input = req.getInputStream();
ReadListenerImpl listener = new ReadListenerImpl( input, 
asyncContext, req, resp, map, myExecutor, this );
input.setReadListener( listener );
. . .
public class ReadListenerImpl implements ReadListener
{
.
.
.
@Override
public void onDataAvailable() throws IOException
{
int len = -1;
byte b[] = new byte[4 * 1024];
while( input.isReady() && 
!input.isFinished() )
{
len = input.read( b );
if( len > 0 )
{
String data = new 
String( b, 0, len );
sb.append( data );
}
}
}

@Override
public void onError(Throwable throwable)
{
asyncContext.complete();
throwable.printStackTrace();

}
@Override
public void onAllDataRead() throws IOException
{   
String jsonData = sb.toString();
.
.
.
}
}

Unfortunately I am still not able to reproduce this exception with a test code 
, I am trying .

Thanks and Regards
Arshiya Shariff


-Original Message-
From: Christopher Schultz  
Sent: Wednesday, October 21, 2020 7:53 PM
To: users@tomcat.apache.org
Subject: Re: FW: HTTP2: memory filled up fast on increasing the connections to 
1000/2000 (Embedded tomcat 9.0.38)

Arshiya,

On 10/21/20 00:34, Arshiya Shariff wrote:
> Hi,
> 
> Christopher, Please find the answer in-line:
> How... exactly?
> private String getRequestBody(HttpServletRequest request) throws IOException
>   {
>   StringBuilder sb = new StringBuilder();
>   BufferedReader reader = request.getReader();
>   try
>   {
>   String line;
>   while( ( line = reader.readLine() ) != null )
>   {
>   sb.append( line ).append( '\n' );

Note that this may modify the incoming request. Are you sure you want to return 
a value which does not match the exact POST body?

>   }
>   }
>   finally
>   {
>   reader.close();
>   }
>   return sb.toString();
>   }   
>  

Is that method run from within the asynchronous context or before you begin 
async processing? I'm not an expert at servlet-async, but I think you should be 
reading the request entirely before entering the async context. Reading the 
request from async may cause problems.

Instead of using blocking reads of the request bbody in asynchronous mode, you 
should do this:

request.getInputStream().setReadListener(new ReadListener() {
  public void onoAllDataRead() { ... }
  public void onDataAvailable() { ... }
  public void onError(Throwable t) { ... } });


> I am trying to reproduce the StackOverflowError with a sample 
> application , Once it is reproduced I will share it across.

See https://www.slideshare.net/SimoneBordet/servlet-31-async-io

Specifically slides 43-53.

-chris

> -Original Message-
> From: Christopher Schultz 
> Sent: Thursday, October 15, 2020 12:01 AM
> To: users@tomcat.apache.org
> Subject: Re: FW: HTTP2: memory filled up fast on increasing the 
> connections to 1000/2000 (Embedded tomcat 9.0.38)
> 
> Arshiya,
> 
> On 10/14/20 01:23, Arshiya Shariff wrote:
>> Please find the answers in-line Mark.
>>
>> Http2 requests with message payload of  34KB are pumped from JMeter 
>> at
>> 20 TPS with 700 connections to an application with

RE: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Eric Robinson 
> On 27/10/2020 09:16, Mark Thomas wrote:
> > On 27/10/2020 04:43, Eric Robinson wrote:
> >
> > 
> >
> >> Any changes in the Nginx configuration in the relevant timescale?
> >>
> 
>  The last change to the nginx config files was on 8/21. The first
>  report of problems from the users in question was on 9/16. There is
>  another set of users on a different tomcat instance who reported
>  issues around 8/26, 5 days after nginx config change. It seems
>  unlikely to be related. Also, I can't imagine what nginx could be
>  sending that would induce the upstream tomcat to behave this way.
> >
> > If there is some sort of retaining references to request/response/etc.
> > at the root of this then that sort of issue is very sensitive to timing.
> > For example, I have had reliable reproduction cases in the past that
> > stopped working with the addition of a single debug statement. Any
> > sort of configuration change might have changed the timing
> > sufficiently to trigger the issue
> >
> > At this point, I'd say the Nginx config change might be a potential
> > trigger if the root cause is retaining references.
> >
> >> Any updates to the application in the relevant timescale?
> >>
> 
>  Their application was patched to a newer version on 6/5.
> >
> > That seems far enough away to be unlikely.
> >
> >> Any features users started using that hadn't been used before in
> >> that timescale?
> 
>  That one I couldn't answer, as we are only the hosting facility and
>  we are not in the loop when it comes to the users' workflow, but it
>  seems unlikely given the nature of their business.
> >
> > Fair enough. That one was a bit of a shot in the dark.
> >
> > 
> >
>  1. Now that you have provided this patch, should I still enable
>  RECYCLE_FACADES=true?
> >
> > I'd recommend yes. At least until the issue is resolved.
> >
>  2. The servers in question are multi-tenanted. There are 17
>  instances of tomcat, each running on a different set of ports and
>  out of a separate directory tree, but they are all started with
>  essentially the same init script, which exports certain
>  site-specific path variables and executes tomcat from the specified
>  folder structure. Can you think of any potential issues where
>  making this change for one instance could have a negative effect on
>  any of the other instances? Probably not, but just being careful. I
>  will have these changes implemented during our nightly maintenance
>  window will begin to gather relevant logs
> >>> first thing tomorrow!
> >
> > I can't think of any side effects.
> >
> 
>  --Eric
> >>>
> >>> Mark, the changes have been made per your instructions and tomcat
> >>> has been restarted. The debug0.log, and debug0.log.lck files were
> >>> created in the directory, but they currently both have 0 bytes.
> >
> > Drat. That suggests something isn't quite right as the logs should
> > start filling up as soon as a single request is made. I'll double
> > check my instructions if you could double check your end.
>
> I've clarified a few things in the instructions and confirmed they work with
> my local 7.0.72 build.
>
> Note: You will need to be using the BIO connector
>

I had switched to the NIO connector last week. Is that why the logs are still 
at 0 bytes?


> Mark
>
> >
> > Konstantin noted there was no source provided. I've pushed the branch
> > to
> > https://github.com/markt-asf/tomcat/tree/debug-7.0.72 so you can see
> > the changes I made.
> >
> >> Also, RECYCLE_FACADES has been enabled and I confirmed that it is
> referenced in the logs as follows...
> >>
> >> INFO: Command line argument:
> >> -Dorg.apache.catalina.connector.RECYCLE_FACADES=true
> >
> > Great.
> >
> > Mark
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional

Re: Default Application questions

2020-10-27 Thread Mark Thomas 
On 26/10/2020 19:37, jonmcalexan...@wellsfargo.com.INVALID wrote:
> I'm doing some documentation cleanup. When was the balancer app, if it 
> existed, removed from the ootb tomcat applications?

It was in 5.0.x (5.0.15 onwards) and 5.5.x.
http://tomcat.apache.org/tomcat-5.5-doc/balancer-howto.html

> Was there ever a webdav app, or just the WebdavServlet class?

Yes there was a webdav app. It was present in 4.0.x, 4.1.x, 5.0.x, and
5.5.x.

I couldn't find a reference in the change log for their removal.

Mark


> 
> Thanks,
> 
> 
> Dream * Excel * Explore * Inspire
> Jon McAlexander
> Infrastructure Engineer
> Asst Vice President
> 
> Middleware Product Engineering
> Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions
> 
> 8080 Cobblestone Rd | Urbandale, IA 50322
> MAC: F4469-010
> Tel 515-988-2508 | Cell 515-988-2508
> 
> jonmcalexan...@wellsfargo.com
> 
> 
> This message may contain confidential and/or privileged information. If you 
> are not the addressee or authorized to receive this for the addressee, you 
> must not use, copy, disclose, or take any action based on this message or any 
> information herein. If you have received this message in error, please advise 
> the sender immediately by reply e-mail and delete this message. Thank you for 
> your cooperation.
> 
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas 
On 27/10/2020 09:16, Mark Thomas wrote:
> On 27/10/2020 04:43, Eric Robinson wrote:
> 
> 
> 
>> Any changes in the Nginx configuration in the relevant timescale?
>>

 The last change to the nginx config files was on 8/21. The first
 report of problems from the users in question was on 9/16. There is
 another set of users on a different tomcat instance who reported
 issues around 8/26, 5 days after nginx config change. It seems
 unlikely to be related. Also, I can't imagine what nginx could be
 sending that would induce the upstream tomcat to behave this way.
> 
> If there is some sort of retaining references to request/response/etc.
> at the root of this then that sort of issue is very sensitive to timing.
> For example, I have had reliable reproduction cases in the past that
> stopped working with the addition of a single debug statement. Any sort
> of configuration change might have changed the timing sufficiently to
> trigger the issue
> 
> At this point, I'd say the Nginx config change might be a potential
> trigger if the root cause is retaining references.
> 
>> Any updates to the application in the relevant timescale?
>>

 Their application was patched to a newer version on 6/5.
> 
> That seems far enough away to be unlikely.
> 
>> Any features users started using that hadn't been used before in
>> that timescale?

 That one I couldn't answer, as we are only the hosting facility and we
 are not in the loop when it comes to the users' workflow, but it seems
 unlikely given the nature of their business.
> 
> Fair enough. That one was a bit of a shot in the dark.
> 
> 
> 
 1. Now that you have provided this patch, should I still enable
 RECYCLE_FACADES=true?
> 
> I'd recommend yes. At least until the issue is resolved.
> 
 2. The servers in question are multi-tenanted. There are 17 instances
 of tomcat, each running on a different set of ports and out of a
 separate directory tree, but they are all started with essentially the
 same init script, which exports certain site-specific path variables
 and executes tomcat from the specified folder structure. Can you think
 of any potential issues where making this change for one instance
 could have a negative effect on any of the other instances? Probably
 not, but just being careful. I will have these changes implemented
 during our nightly maintenance window will begin to gather relevant logs
>>> first thing tomorrow!
> 
> I can't think of any side effects.
> 

 --Eric
>>>
>>> Mark, the changes have been made per your instructions and tomcat has
>>> been restarted. The debug0.log, and debug0.log.lck files were created in the
>>> directory, but they currently both have 0 bytes.
> 
> Drat. That suggests something isn't quite right as the logs should start
> filling up as soon as a single request is made. I'll double check my
> instructions if you could double check your end.

I've clarified a few things in the instructions and confirmed they work
with my local 7.0.72 build.

Note: You will need to be using the BIO connector

Mark

> 
> Konstantin noted there was no source provided. I've pushed the branch to
> https://github.com/markt-asf/tomcat/tree/debug-7.0.72 so you can see the
> changes I made.
> 
>> Also, RECYCLE_FACADES has been enabled and I confirmed that it is referenced 
>> in the logs as follows...
>>
>> INFO: Command line argument: 
>> -Dorg.apache.catalina.connector.RECYCLE_FACADES=true
> 
> Great.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Weirdest Tomcat Behavior Ever?

2020-10-27 Thread Mark Thomas 
On 27/10/2020 04:43, Eric Robinson wrote:



> Any changes in the Nginx configuration in the relevant timescale?
>
>>>
>>> The last change to the nginx config files was on 8/21. The first
>>> report of problems from the users in question was on 9/16. There is
>>> another set of users on a different tomcat instance who reported
>>> issues around 8/26, 5 days after nginx config change. It seems
>>> unlikely to be related. Also, I can't imagine what nginx could be
>>> sending that would induce the upstream tomcat to behave this way.

If there is some sort of retaining references to request/response/etc.
at the root of this then that sort of issue is very sensitive to timing.
For example, I have had reliable reproduction cases in the past that
stopped working with the addition of a single debug statement. Any sort
of configuration change might have changed the timing sufficiently to
trigger the issue

At this point, I'd say the Nginx config change might be a potential
trigger if the root cause is retaining references.

> Any updates to the application in the relevant timescale?
>
>>>
>>> Their application was patched to a newer version on 6/5.

That seems far enough away to be unlikely.

> Any features users started using that hadn't been used before in
> that timescale?
>>>
>>> That one I couldn't answer, as we are only the hosting facility and we
>>> are not in the loop when it comes to the users' workflow, but it seems
>>> unlikely given the nature of their business.

Fair enough. That one was a bit of a shot in the dark.



>>> 1. Now that you have provided this patch, should I still enable
>>> RECYCLE_FACADES=true?

I'd recommend yes. At least until the issue is resolved.

>>> 2. The servers in question are multi-tenanted. There are 17 instances
>>> of tomcat, each running on a different set of ports and out of a
>>> separate directory tree, but they are all started with essentially the
>>> same init script, which exports certain site-specific path variables
>>> and executes tomcat from the specified folder structure. Can you think
>>> of any potential issues where making this change for one instance
>>> could have a negative effect on any of the other instances? Probably
>>> not, but just being careful. I will have these changes implemented
>>> during our nightly maintenance window will begin to gather relevant logs
>> first thing tomorrow!

I can't think of any side effects.

>>>
>>> --Eric
>>
>> Mark, the changes have been made per your instructions and tomcat has
>> been restarted. The debug0.log, and debug0.log.lck files were created in the
>> directory, but they currently both have 0 bytes.

Drat. That suggests something isn't quite right as the logs should start
filling up as soon as a single request is made. I'll double check my
instructions if you could double check your end.

Konstantin noted there was no source provided. I've pushed the branch to
https://github.com/markt-asf/tomcat/tree/debug-7.0.72 so you can see the
changes I made.

> Also, RECYCLE_FACADES has been enabled and I confirmed that it is referenced 
> in the logs as follows...
> 
> INFO: Command line argument: 
> -Dorg.apache.catalina.connector.RECYCLE_FACADES=true

Great.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org