Re: BasicDataSource restart()

[Mark Eggers]

Jerry,

On 11/30/2021 10:06 PM, Jerry Malcolm wrote:
I'm circling back to this after a few months.  I am building on a 
Windows 10 box with 9.0.52 TC.  But I'm running on an AWS Linux2 with 
8.5.72.  This has never caused any problems so far, or at least as far 
as I can tell.  But I'm hitting something strange with this relatively 
new BasicDataSource.restart() method.   My reference to restart() builds 
fine.  But when I run it on the 8.5.72 server I get: 
java.lang.NoSuchMethodError: 'void 
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.restart()'


Rémy mentioned that it should be in TC 8.5.58+.   This is a relatively 
clean EC2 on AWS, not running much other than Tomcat. And I have not 
done in backdoor (non yum) installs of TC that might have left old jar 
files around.   I noticed that there are 2 dbcp2 jar files in TC's lib. 
One is from the java install.  But the error message above is a tomcat 
path.  So I'm assuming it's tomcat's dbcp2 jar that's being referenced. 
I exploded the jar hoping there would be some version numbers somewhere 
inside telling me if I somehow have a backlevel jar.  But I couldn't 
find anything.  All I know is it's date is 2/25/2021 and it's 286,358 
bytes.


Any other ideas come to mind why it's telling me the restart() method 
doesn't exist?


Thx as always.

Jerry


On 9/7/2021 2:49 PM, Jerry Malcolm wrote:


On 9/7/2021 2:35 PM, Christopher Schultz wrote:

Jerry, Rémy,

On 9/3/21 07:15, Rémy Maucherat wrote:
On Fri, Sep 3, 2021 at 2:46 AM Jerry Malcolm 
 wrote:


I have a requirement to start a new log database on the first of every
month.  I still need to have access to older monthly log 
databases.   I

do not want to create a bunch of hardcoded manually configured
individual datasources, one for each month.  I have a dynamic 
datasource

solution that is completely implemented and working except for one
little thing.

I access the BasicDataSource implementation class for the 
datasource.  I

have an algorithm that substitutes _MM at the appropriate spot in
the configured URL and then updates the url in the datasource. All of
this works great.  I can live with the fact that the datasource can 
only

point to one database at a time.  My concern is that once I transition
to another database, there are existing connections in the pool 
that are

already attached to the old database.  I need to clear those out and
start over.  But I don't have the luxury of bouncing tomcat to 
clean it up.


The apache commons BasicDataSource has a restart() method. But
unfortunately that method is omitted from the Tomcat version. There 
is a

close() method on the BasicDataSource.  But I don't see anything that
will re-open it after closing.  I thought about changing maxActive 
to 0,

and waiting for it to drain, then setting it back to the original
value.  None of these sound like an ideal solution.  Without a 
restart()

method, is there any other way to force all existing connections to
close and start clean?


The code is kept in sync with DBCP (with a bit of lag maybe), so these
lifecycle methods were also added to Tomcat one year ago (9.0.38+ and
8.5.58+).


We are using this at $work to bounce our database connection pools 
after TLS client certificate changes. This is the code we are using 
to reload the pool:


  try
  {
  Context ctx = new InitialContext();

  DataSource ds = (DataSource)ctx.lookup(getJNDIPath());

  if(null == ds)
  throw new ServiceException("Cannot obtain DataSource");

  if(isInstanceOf(ds, 
"org.apache.tomcat.dbcp.dbcp2.BasicDataSource")
 || isInstanceOf(ds, 
"org.apache.commons.dbcp2.BasicDataSource")) {


  return call(ds, "restart");
  }
  } catch (Exception e) {
org.apache.log4j.Logger.getLogger(this.getClass()).error("Failed to 
reload DataSource " + getJNDIPath());

  }

The call() method simply encapsulates all of the work to make a 
reflective method call to BasicDataSource.restart().


As Rémy points out, it requires a Tomcat version 9.0.38+ or 8.5.58+.

Hope that helps,
-chris


Chris,

I'll definitely try this.  But I'm curious about the restart method. 
Is it some sort of a hidden method that's only available to 
reflection?  Seems it would be a lot more straightforward to just make 
the restart method public like it is in the apache version of 
BasicDataSource.  I'm not complaining.  If this works, then fine. Just 
curious.


Thx

Jerry


I haven't been following this thread, so I may be way off.

The last time I used an AWS EC2 instance "out of the box" with an 
AWS-supplied Tomcat, I ran into some very strange behavior.


It turns out that AWS packaged the 8.5.x Tomcat with the older (7.0.x) 
resource pool. I figured this out by looking at logs and seeing the 
complaints about my context.xml.


I raised the issue with AWS, and got silence back.

Ever since then, I package up my own version of Tomcat using releases 
from tomcat.apache.org.


Could you be experiencing some similar issue?

. . .

Re: BasicDataSource restart()

[Jerry Malcolm]

I'm circling back to this after a few months.  I am building on a 
Windows 10 box with 9.0.52 TC.  But I'm running on an AWS Linux2 with 
8.5.72.  This has never caused any problems so far, or at least as far 
as I can tell.  But I'm hitting something strange with this relatively 
new BasicDataSource.restart() method.   My reference to restart() builds 
fine.  But when I run it on the 8.5.72 server I get:  
java.lang.NoSuchMethodError: 'void 
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.restart()'


Rémy mentioned that it should be in TC 8.5.58+.   This is a relatively 
clean EC2 on AWS, not running much other than Tomcat. And I have not 
done in backdoor (non yum) installs of TC that might have left old jar 
files around.   I noticed that there are 2 dbcp2 jar files in TC's lib.  
One is from the java install.  But the error message above is a tomcat 
path.  So I'm assuming it's tomcat's dbcp2 jar that's being referenced.  
I exploded the jar hoping there would be some version numbers somewhere 
inside telling me if I somehow have a backlevel jar.  But I couldn't 
find anything.  All I know is it's date is 2/25/2021 and it's 286,358 bytes.


Any other ideas come to mind why it's telling me the restart() method 
doesn't exist?


Thx as always.

Jerry


On 9/7/2021 2:49 PM, Jerry Malcolm wrote:


On 9/7/2021 2:35 PM, Christopher Schultz wrote:

Jerry, Rémy,

On 9/3/21 07:15, Rémy Maucherat wrote:
On Fri, Sep 3, 2021 at 2:46 AM Jerry Malcolm 
 wrote:


I have a requirement to start a new log database on the first of every
month.  I still need to have access to older monthly log 
databases.   I

do not want to create a bunch of hardcoded manually configured
individual datasources, one for each month.  I have a dynamic 
datasource

solution that is completely implemented and working except for one
little thing.

I access the BasicDataSource implementation class for the 
datasource.  I

have an algorithm that substitutes _MM at the appropriate spot in
the configured URL and then updates the url in the datasource.   
All of
this works great.  I can live with the fact that the datasource can 
only

point to one database at a time.  My concern is that once I transition
to another database, there are existing connections in the pool 
that are

already attached to the old database.  I need to clear those out and
start over.  But I don't have the luxury of bouncing tomcat to 
clean it up.


The apache commons BasicDataSource has a restart() method. But
unfortunately that method is omitted from the Tomcat version. There 
is a

close() method on the BasicDataSource.  But I don't see anything that
will re-open it after closing.  I thought about changing maxActive 
to 0,

and waiting for it to drain, then setting it back to the original
value.  None of these sound like an ideal solution.  Without a 
restart()

method, is there any other way to force all existing connections to
close and start clean?


The code is kept in sync with DBCP (with a bit of lag maybe), so these
lifecycle methods were also added to Tomcat one year ago (9.0.38+ and
8.5.58+).


We are using this at $work to bounce our database connection pools 
after TLS client certificate changes. This is the code we are using 
to reload the pool:


  try
  {
  Context ctx = new InitialContext();

  DataSource ds = (DataSource)ctx.lookup(getJNDIPath());

  if(null == ds)
  throw new ServiceException("Cannot obtain DataSource");

  if(isInstanceOf(ds, 
"org.apache.tomcat.dbcp.dbcp2.BasicDataSource")
 || isInstanceOf(ds, 
"org.apache.commons.dbcp2.BasicDataSource")) {


  return call(ds, "restart");
  }
  } catch (Exception e) {
org.apache.log4j.Logger.getLogger(this.getClass()).error("Failed to 
reload DataSource " + getJNDIPath());

  }

The call() method simply encapsulates all of the work to make a 
reflective method call to BasicDataSource.restart().


As Rémy points out, it requires a Tomcat version 9.0.38+ or 8.5.58+.

Hope that helps,
-chris


Chris,

I'll definitely try this.  But I'm curious about the restart method.  
Is it some sort of a hidden method that's only available to 
reflection?  Seems it would be a lot more straightforward to just make 
the restart method public like it is in the apache version of 
BasicDataSource.  I'm not complaining.  If this works, then fine.  
Just curious.


Thx

Jerry


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: NPE in request.getRequestURL()

[Jerry Malcolm]

On 11/30/2021 1:58 PM, Christopher Schultz wrote:

Jerry,

On 11/30/21 14:17, Jerry Malcolm wrote:

Chris, Thanks for the response.

Sorry... forgot to include the TC ver -- 8.5.69.

I had a situation a while back where I spun a longrunning thread and 
held the request object after the main response was returned. I fixed 
that situation.  In this situation, it is occurring on the main 
request/response thread.  Is there any situation where the request 
object could be recycled before the associated response is returned?


Not usually. So the thread hitting the NPE is something like 
[catalina-exec- ...]?


And you are sure you aren't doing anything funny with the request 
object? No storing it in the session or getting a reference from some 
cross-thread storage mechanism?


Looking at the code 
(https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/catalina/connector/Request.java#L2465):


    @Override
    public StringBuffer getRequestURL() {

    StringBuffer url = new StringBuffer();
    String scheme = getScheme();
    int port = getServerPort();
    if (port < 0)
 {
    port = 80; // Work around java.net.URL bug
    }

    url.append(scheme);
    url.append("://");
    url.append(getServerName());
    if ((scheme.equals("http") && (port != 80))
    || (scheme.equals("https") && (port != 443))) {
    url.append(':'); //  This is line 2465
    url.append(port);
    }
    url.append(getRequestURI());

    return url;
    }

I don't see how an NPE could happen on that line.

But 8.5.69 was released on 2021-07-05, and that code is here:

https://github.com/apache/tomcat/blob/3e9dd49b20f9d6e270f8709d4f16d5595977595e/java/org/apache/catalina/connector/Request.java 



This makes a little more sense:

    @Override
    public StringBuffer getRequestURL() {

    StringBuffer url = new StringBuffer();
    String scheme = getScheme();
    int port = getServerPort();
    if (port < 0)
 {
    port = 80; // Work around java.net.URL bug
    }

    url.append(scheme);
    url.append("://");
    url.append(getServerName());
    if ((scheme.equals("http") && (port != 80)) // <<< This is 2465
    || (scheme.equals("https") && (port != 443))) {
    url.append(':');
    url.append(port);
    }
    url.append(getRequestURI());

    return url;
    }

So the scheme is null. That's odd, since getScheme is:

    public String getScheme() {
    return coyoteRequest.scheme().toString();
    }

Oh, but coyoteRequest.scheme() returns a MessageBytes object whose 
toString method can (somewhat surprisingly) return null(!) when the 
type is T_NULL. And wouldn't you know it, here is the code for 
MessageBytes.recycle() (which gets called whenever the request, and 
therefore all of the various parts of the request, is recycled):


    public void recycle() {
    type=T_NULL;
    byteC.recycle();
    charC.recycle();

    strValue=null;

    hasStrValue=false;
    hasHashCode=false;
    hasLongValue=false;
    }

So it definitely looks like your request has been recycled somehow.

-chris

Chris, I was running 8.5.69.  (I just bumped to 8.5.72).  So I agree 
that everything looks like a recycled request.  This is a REST api.  
Pretty much come in, do the work, return.  The request object is carried 
around throughout the api in a 'briefcase' object.  But nothing is 
cached in the session object or anywhere. Pretty much a new sunrise on 
every api/servlet call coming in.


Just to confirm my understanding of request recycling, a request object 
is assigned before the request is handed to my handler servlet.  When 
the servlet exits, the request object is recycled and returned to the 
pool.   For this problem to be a recycled request problem, it would mean 
that I saved off a reference to a request and referenced it on a 
different servlet call.  It would have to be on a different call, since 
the service of the api is a single thread in and out.  I've been around 
too long to say that it couldn't happen.  But at this point, I'm 
struggling with how. It would mean I would have to discard the correct 
request for the servlet call and replace it with an old version.  I'm 
going to have add some logging statements.  Is there a unique id or 
something for each request object in the pool that I might be able to 
get access to?  If so, I can start logging the request UID at the 
beginning of the servlet call and log the UID on each access to the 
request object.


Let me know if there's some magic ID I can log to try to figure out when 
i might be using a recycled instance?


Thx

Jerry




On 11/29/2021 7:42 PM, Christopher Schultz wrote:

Jerry,

On 11/29/21 19:33, Jerry Malcolm wrote:
Can anyone tell me what I might be doing that would cause an NPE 
inside the request object on getRequestURL()?  The NPE only happens 
about

Re: NPE in request.getRequestURL()

[Christopher Schultz]

Jerry,

On 11/30/21 14:17, Jerry Malcolm wrote:

Chris, Thanks for the response.

Sorry... forgot to include the TC ver -- 8.5.69.

I had a situation a while back where I spun a longrunning thread and 
held the request object after the main response was returned. I fixed 
that situation.  In this situation, it is occurring on the main 
request/response thread.  Is there any situation where the request 
object could be recycled before the associated response is returned?


Not usually. So the thread hitting the NPE is something like 
[catalina-exec- ...]?


And you are sure you aren't doing anything funny with the request 
object? No storing it in the session or getting a reference from some 
cross-thread storage mechanism?


Looking at the code 
(https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/catalina/connector/Request.java#L2465):


@Override
public StringBuffer getRequestURL() {

StringBuffer url = new StringBuffer();
String scheme = getScheme();
int port = getServerPort();
if (port < 0)
 {
port = 80; // Work around java.net.URL bug
}

url.append(scheme);
url.append("://");
url.append(getServerName());
if ((scheme.equals("http") && (port != 80))
|| (scheme.equals("https") && (port != 443))) {
url.append(':'); //  This is line 2465
url.append(port);
}
url.append(getRequestURI());

return url;
}

I don't see how an NPE could happen on that line.

But 8.5.69 was released on 2021-07-05, and that code is here:

https://github.com/apache/tomcat/blob/3e9dd49b20f9d6e270f8709d4f16d5595977595e/java/org/apache/catalina/connector/Request.java

This makes a little more sense:

@Override
public StringBuffer getRequestURL() {

StringBuffer url = new StringBuffer();
String scheme = getScheme();
int port = getServerPort();
if (port < 0)
 {
port = 80; // Work around java.net.URL bug
}

url.append(scheme);
url.append("://");
url.append(getServerName());
if ((scheme.equals("http") && (port != 80)) // <<< This is 2465
|| (scheme.equals("https") && (port != 443))) {
url.append(':');
url.append(port);
}
url.append(getRequestURI());

return url;
}

So the scheme is null. That's odd, since getScheme is:

public String getScheme() {
return coyoteRequest.scheme().toString();
}

Oh, but coyoteRequest.scheme() returns a MessageBytes object whose 
toString method can (somewhat surprisingly) return null(!) when the type 
is T_NULL. And wouldn't you know it, here is the code for 
MessageBytes.recycle() (which gets called whenever the request, and 
therefore all of the various parts of the request, is recycled):


public void recycle() {
type=T_NULL;
byteC.recycle();
charC.recycle();

strValue=null;

hasStrValue=false;
hasHashCode=false;
hasLongValue=false;
}

So it definitely looks like your request has been recycled somehow.

-chris


On 11/29/2021 7:42 PM, Christopher Schultz wrote:

Jerry,

On 11/29/21 19:33, Jerry Malcolm wrote:
Can anyone tell me what I might be doing that would cause an NPE 
inside the request object on getRequestURL()?  The NPE only happens 
about 10% of the time.  I can't figure out what is happening 
differently in that 10% of the calls to this code.  Thx


java.lang.NullPointerException
 at 
org.apache.catalina.connector.Request.getRequestURL(Request.java:2465)
 at 
org.apache.catalina.connector.RequestFacade.getRequestURL(RequestFacade.java:868) 




Thoughts:

1. Tomcat version?
2. Retained reference to request object that has been recycled?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9 doesn't shutdown cleanly

[Terence M. Bandoian]

On 11/30/2021 8:40 AM, Simon Matter wrote:

Hi Chris,

Thank you for the quick reply.


Simon,

On 11/30/21 08:21, Simon Matter wrote:

I'm running an application on Tomcat 9.0.55 on x86_64 Linux with OpenJDK
JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.

When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly
(most
threads are gone) but send a thread dump to catalina.out and is later
killer by an OS signal.

This should only happen if you have CATALINA_PID defined. Are you indeed
defining that environment variable?

catalina.sh has this code in it, which is probably what you are seeing:

echo "To aid diagnostics a thread dump has been written to
standard out."
kill -3 `cat "$CATALINA_PID"`


That's true, I have CATALINA_PID defined when the call of "catalina.sh
start" is done. So yes, the script will kill the java process if it
doesn't terminate.


When shutting down Tomcat via the shutdown listener on port 8005, it
also
shuts down mostly but without the thread dump in catalina.out. Sending
SIGTERM later to the still running java process terminates it.

Right: when you manually connect to the shutdown port and send
"SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't send a
signal. You have to do that manually, too.

On a test host, when I send "SHUTDOWN" to the shutdown listener, Tomcat
shuts down and the Java VM terminates.

Only on this host with the application, when I send "SHUTDOWN" to the
shutdown listener, Tomcat shuts down but the Java VM doesn't terminate.


So both methods somehow terminate Tomcat partly but not completely.

You have one or two of the following issues:

1. You have a non-daemon thread running
2. You have an unusually long shutdown process that just takes "too long"

The good news is that the thread-dup can answer that question: what's in
the thread dump that is generated when you run "catalina.sh stop"?


When simply sending SIGTERM on the OS level, Tomcat shuts down cleanly
and
terminates without issues.

One thing in common is that I always see these messages while shutting
down:

30-Nov-2021 13:59:36.985 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class [com.sun.proxy.$Proxy51] and
value
[Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastRef
[liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7ffd,
-4005526521234186948]]. This RMI Target has been forcibly removed to
prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.

Why do the three methods to shutdown Tomcat differ and how can I make
'catalina.sh stop' or the shutdown listener work the same way like
sending
the OS signal.

What's the difference? You don't want the thread dump in your
catalina.out? That's supposed to be helpful in debugging why your
shutdown isn't clean. It sounds like you are saying "help me with my
unclean shutdown; please get rid of the debugging information that is
trying to help me!".


I've tried, beside a lot of other things, to set
skipMemoryLeakChecksOnJvmShutdown="true" in context.xml but it seems to
change nothing at all.

Tomcat will never kill your non-daemon thread(s) from inside the VM.
Since you are using a CATALINA_PID environment variable (and, therefore,
a pid-file), the shutdown process is sending the TERM signal. It's also
possibly sending a KILL signal, depending on whether or not the TERM
worked and if -force was supplied on the command-line.

Unfortunately I am not an insider when it comes to Java, so I'm not sure
what a non-daemon thread means here? Can these be threads started by our
application which are non-daemon threads? So they will be terminated with
SIGTERM to the Java VM but not using the shutdown listener?

I'll paste a thread dump below.

Thanks,
Simon

Full thread dump OpenJDK 64-Bit Server VM (11.0.13+8 mixed mode):

Threads class SMR info:
_java_thread_list=0x55a49ac4ba60, length=18, elements={
0x7f1ac0148800, 0x7f1ac014a800, 0x7f1ac015f800,
0x7f1ac0161800,
0x7f1ac0164000, 0x7f1ac0168800, 0x7f1ac016a000,
0x7f1ac01ad800,
0x7f1ac0258000, 0x7f1ac0311000, 0x7f1ac0318000,
0x7f1ac031a000,
0x7f1ac25d6800, 0x7f1ac25fe800, 0x

Re: NPE in request.getRequestURL()

[Jerry Malcolm]

Chris, Thanks for the response.

Sorry... forgot to include the TC ver -- 8.5.69.

I had a situation a while back where I spun a longrunning thread and 
held the request object after the main response was returned. I fixed 
that situation.  In this situation, it is occurring on the main 
request/response thread.  Is there any situation where the request 
object could be recycled before the associated response is returned?


Thx

Jerry

On 11/29/2021 7:42 PM, Christopher Schultz wrote:

Jerry,

On 11/29/21 19:33, Jerry Malcolm wrote:
Can anyone tell me what I might be doing that would cause an NPE 
inside the request object on getRequestURL()?  The NPE only happens 
about 10% of the time.  I can't figure out what is happening 
differently in that 10% of the calls to this code.  Thx


java.lang.NullPointerException
 at 
org.apache.catalina.connector.Request.getRequestURL(Request.java:2465)
 at 
org.apache.catalina.connector.RequestFacade.getRequestURL(RequestFacade.java:868) 



Thoughts:

1. Tomcat version?
2. Retained reference to request object that has been recycled?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

AW: AW: Tomcat 9 doesn't shutdown cleanly

[Thomas Hoffmann (Speed4Trade GmbH)]

Hello Simon,

if you use the Registry to bind Objects / Stubs, you must also call "unbind" on 
shutdown:
https://docs.oracle.com/javase/7/docs/api/java/rmi/registry/Registry.html

I think the developer who implemented the RMI stub, should also now what to 
unbind.

Greetings,
Thomas

-Ursprüngliche Nachricht-
Von: Simon Matter  
Gesendet: Dienstag, 30. November 2021 16:10
An: Tomcat Users List 
Betreff: Re: AW: Tomcat 9 doesn't shutdown cleanly

Hi Thomas,

> Hello,
>
> it looks like your application opens several ports for RMI communication.
> One class is mentioned in your first mail: ShopdbCacheSynchronizer

Yes, and it's true that when I'm looking at the shut down Tomcat instance VM, I 
see several RMI threads lingering around.

>
> Maybe you can ask the developer guys to check whether these threads / 
> ports are terminated / closed cleanly on shutdown event.
> Quite often developers don't care much about shutting down their stuff 
> cleanly.

Good guess, that's exactly what I'm trying to do, finding out why we have the 
shutdown problems so the "developer guys" can finally fix this issue.

From how I understand it, we have an internal application server which 
initiates RMI connections to the Tomcat instance sitting in the DMZ. My 
question is, can we terminate the RMI connections on the Tomcat instance only 
and shut down Tomcat, or do we have to close the RMI connections on the 
internal appserver which initiated the connections?

Apart from RMI, is there anything in the thread dump which indicates an issue 
in out Tomcat app?

Kind regards,
Simon

>
> Greetings,
> Thomas
>
> -Ursprüngliche Nachricht-
> Von: Simon Matter 
> Gesendet: Dienstag, 30. November 2021 15:40
> An: Tomcat Users List 
> Betreff: Re: Tomcat 9 doesn't shutdown cleanly
>
> Hi Chris,
>
> Thank you for the quick reply.
>
>> Simon,
>>
>> On 11/30/21 08:21, Simon Matter wrote:
>>> I'm running an application on Tomcat 9.0.55 on x86_64 Linux with 
>>> OpenJDK
>>> JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.
>>>
>>> When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly 
>>> (most threads are gone) but send a thread dump to catalina.out and 
>>> is later killer by an OS signal.
>>
>> This should only happen if you have CATALINA_PID defined. Are you 
>> indeed defining that environment variable?
>>
>> catalina.sh has this code in it, which is probably what you are seeing:
>>
>>echo "To aid diagnostics a thread dump has been written to 
>> standard out."
>>kill -3 `cat "$CATALINA_PID"`
>>
>
> That's true, I have CATALINA_PID defined when the call of "catalina.sh 
> start" is done. So yes, the script will kill the java process if it 
> doesn't terminate.
>
>>> When shutting down Tomcat via the shutdown listener on port 8005, it 
>>> also shuts down mostly but without the thread dump in catalina.out.
>>> Sending SIGTERM later to the still running java process terminates 
>>> it.
>>
>> Right: when you manually connect to the shutdown port and send 
>> "SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't 
>> send a signal. You have to do that manually, too.
>
> On a test host, when I send "SHUTDOWN" to the shutdown listener, 
> Tomcat shuts down and the Java VM terminates.
>
> Only on this host with the application, when I send "SHUTDOWN" to the 
> shutdown listener, Tomcat shuts down but the Java VM doesn't terminate.
>
>>
>>> So both methods somehow terminate Tomcat partly but not completely.
>>
>> You have one or two of the following issues:
>>
>> 1. You have a non-daemon thread running 2. You have an unusually long 
>> shutdown process that just takes "too long"
>>
>> The good news is that the thread-dup can answer that question: what's 
>> in the thread dump that is generated when you run "catalina.sh stop"?
>>
>>> When simply sending SIGTERM on the OS level, Tomcat shuts down 
>>> cleanly and terminates without issues.
>>>
>>> One thing in common is that I always see these messages while 
>>> shutting
>>> down:
>>>
>>> 30-Nov-2021 13:59:36.985 SEVERE [main] 
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiT
>>> a rgets Found RMI Target with stub class class 
>>> [sun.rmi.registry.RegistryImpl_Stub] and value 
>>> [RegistryImpl_Stub[UnicastRef [liveRef:
>>> [endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI 
>>> Target has been forcibly removed to prevent a memory leak.
>>> 30-Nov-2021 13:59:36.987 SEVERE [main] 
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiT
>>> a rgets Found RMI Target with stub class class 
>>> [com.sun.proxy.$Proxy51] and value 
>>> [Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[Unicast
>>> R
>>> ef
>>> [liveRef:
>>> [endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-
>>> 7 ffd, -4005526521234186948]]. This RMI Target has been forcibly 
>>> removed to prevent a memory leak.
>>> 30-Nov-2021 13:59:36.987 SEVERE [main] 
>>> org.

Re: AW: Tomcat 9 doesn't shutdown cleanly

[Simon Matter]

Hi Thomas,

> Hello,
>
> it looks like your application opens several ports for RMI communication.
> One class is mentioned in your first mail: ShopdbCacheSynchronizer

Yes, and it's true that when I'm looking at the shut down Tomcat instance
VM, I see several RMI threads lingering around.

>
> Maybe you can ask the developer guys to check whether these threads /
> ports are terminated / closed cleanly on shutdown event.
> Quite often developers don't care much about shutting down their stuff
> cleanly.

Good guess, that's exactly what I'm trying to do, finding out why we have
the shutdown problems so the "developer guys" can finally fix this issue.

>From how I understand it, we have an internal application server which
initiates RMI connections to the Tomcat instance sitting in the DMZ. My
question is, can we terminate the RMI connections on the Tomcat instance
only and shut down Tomcat, or do we have to close the RMI connections on
the internal appserver which initiated the connections?

Apart from RMI, is there anything in the thread dump which indicates an
issue in out Tomcat app?

Kind regards,
Simon

>
> Greetings,
> Thomas
>
> -Ursprüngliche Nachricht-
> Von: Simon Matter 
> Gesendet: Dienstag, 30. November 2021 15:40
> An: Tomcat Users List 
> Betreff: Re: Tomcat 9 doesn't shutdown cleanly
>
> Hi Chris,
>
> Thank you for the quick reply.
>
>> Simon,
>>
>> On 11/30/21 08:21, Simon Matter wrote:
>>> I'm running an application on Tomcat 9.0.55 on x86_64 Linux with
>>> OpenJDK
>>> JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.
>>>
>>> When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly
>>> (most threads are gone) but send a thread dump to catalina.out and is
>>> later killer by an OS signal.
>>
>> This should only happen if you have CATALINA_PID defined. Are you
>> indeed defining that environment variable?
>>
>> catalina.sh has this code in it, which is probably what you are seeing:
>>
>>echo "To aid diagnostics a thread dump has been written to
>> standard out."
>>kill -3 `cat "$CATALINA_PID"`
>>
>
> That's true, I have CATALINA_PID defined when the call of "catalina.sh
> start" is done. So yes, the script will kill the java process if it
> doesn't terminate.
>
>>> When shutting down Tomcat via the shutdown listener on port 8005, it
>>> also shuts down mostly but without the thread dump in catalina.out.
>>> Sending SIGTERM later to the still running java process terminates
>>> it.
>>
>> Right: when you manually connect to the shutdown port and send
>> "SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't send
>> a signal. You have to do that manually, too.
>
> On a test host, when I send "SHUTDOWN" to the shutdown listener, Tomcat
> shuts down and the Java VM terminates.
>
> Only on this host with the application, when I send "SHUTDOWN" to the
> shutdown listener, Tomcat shuts down but the Java VM doesn't terminate.
>
>>
>>> So both methods somehow terminate Tomcat partly but not completely.
>>
>> You have one or two of the following issues:
>>
>> 1. You have a non-daemon thread running 2. You have an unusually long
>> shutdown process that just takes "too long"
>>
>> The good news is that the thread-dup can answer that question: what's
>> in the thread dump that is generated when you run "catalina.sh stop"?
>>
>>> When simply sending SIGTERM on the OS level, Tomcat shuts down
>>> cleanly and terminates without issues.
>>>
>>> One thing in common is that I always see these messages while
>>> shutting
>>> down:
>>>
>>> 30-Nov-2021 13:59:36.985 SEVERE [main]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>>> rgets Found RMI Target with stub class class
>>> [sun.rmi.registry.RegistryImpl_Stub] and value
>>> [RegistryImpl_Stub[UnicastRef [liveRef:
>>> [endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
>>> Target has been forcibly removed to prevent a memory leak.
>>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>>> rgets Found RMI Target with stub class class [com.sun.proxy.$Proxy51]
>>> and value
>>> [Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastR
>>> ef
>>> [liveRef:
>>> [endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7
>>> ffd, -4005526521234186948]]. This RMI Target has been forcibly
>>> removed to prevent a memory leak.
>>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>>> rgets Found RMI Target with stub class class
>>> [sun.rmi.registry.RegistryImpl_Stub] and value
>>> [RegistryImpl_Stub[UnicastRef [liveRef:
>>> [endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI
>>> Target has been forcibly removed to prevent a memory leak.
>>>
>>> Why do the three methods to shutdown Tomcat differ and how can I make
>>> 'catalina.sh stop' or the shutdown listener work the same way like
>>> s

AW: Tomcat 9 doesn't shutdown cleanly

[Thomas Hoffmann (Speed4Trade GmbH)]

Hello,

it looks like your application opens several ports for RMI communication.
One class is mentioned in your first mail: ShopdbCacheSynchronizer

Maybe you can ask the developer guys to check whether these threads / ports are 
terminated / closed cleanly on shutdown event.
Quite often developers don't care much about shutting down their stuff cleanly.

Greetings,
Thomas

-Ursprüngliche Nachricht-
Von: Simon Matter  
Gesendet: Dienstag, 30. November 2021 15:40
An: Tomcat Users List 
Betreff: Re: Tomcat 9 doesn't shutdown cleanly

Hi Chris,

Thank you for the quick reply.

> Simon,
>
> On 11/30/21 08:21, Simon Matter wrote:
>> I'm running an application on Tomcat 9.0.55 on x86_64 Linux with 
>> OpenJDK
>> JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.
>>
>> When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly 
>> (most threads are gone) but send a thread dump to catalina.out and is 
>> later killer by an OS signal.
>
> This should only happen if you have CATALINA_PID defined. Are you 
> indeed defining that environment variable?
>
> catalina.sh has this code in it, which is probably what you are seeing:
>
>echo "To aid diagnostics a thread dump has been written to 
> standard out."
>kill -3 `cat "$CATALINA_PID"`
>

That's true, I have CATALINA_PID defined when the call of "catalina.sh start" 
is done. So yes, the script will kill the java process if it doesn't terminate.

>> When shutting down Tomcat via the shutdown listener on port 8005, it 
>> also shuts down mostly but without the thread dump in catalina.out. 
>> Sending SIGTERM later to the still running java process terminates 
>> it.
>
> Right: when you manually connect to the shutdown port and send 
> "SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't send 
> a signal. You have to do that manually, too.

On a test host, when I send "SHUTDOWN" to the shutdown listener, Tomcat shuts 
down and the Java VM terminates.

Only on this host with the application, when I send "SHUTDOWN" to the shutdown 
listener, Tomcat shuts down but the Java VM doesn't terminate.

>
>> So both methods somehow terminate Tomcat partly but not completely.
>
> You have one or two of the following issues:
>
> 1. You have a non-daemon thread running 2. You have an unusually long 
> shutdown process that just takes "too long"
>
> The good news is that the thread-dup can answer that question: what's 
> in the thread dump that is generated when you run "catalina.sh stop"?
>
>> When simply sending SIGTERM on the OS level, Tomcat shuts down 
>> cleanly and terminates without issues.
>>
>> One thing in common is that I always see these messages while 
>> shutting
>> down:
>>
>> 30-Nov-2021 13:59:36.985 SEVERE [main] 
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>> rgets Found RMI Target with stub class class 
>> [sun.rmi.registry.RegistryImpl_Stub] and value 
>> [RegistryImpl_Stub[UnicastRef [liveRef:
>> [endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI 
>> Target has been forcibly removed to prevent a memory leak.
>> 30-Nov-2021 13:59:36.987 SEVERE [main] 
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>> rgets Found RMI Target with stub class class [com.sun.proxy.$Proxy51] 
>> and value 
>> [Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastR
>> ef
>> [liveRef:
>> [endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7
>> ffd, -4005526521234186948]]. This RMI Target has been forcibly 
>> removed to prevent a memory leak.
>> 30-Nov-2021 13:59:36.987 SEVERE [main] 
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>> rgets Found RMI Target with stub class class 
>> [sun.rmi.registry.RegistryImpl_Stub] and value 
>> [RegistryImpl_Stub[UnicastRef [liveRef:
>> [endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI 
>> Target has been forcibly removed to prevent a memory leak.
>>
>> Why do the three methods to shutdown Tomcat differ and how can I make 
>> 'catalina.sh stop' or the shutdown listener work the same way like 
>> sending the OS signal.
>
> What's the difference? You don't want the thread dump in your 
> catalina.out? That's supposed to be helpful in debugging why your 
> shutdown isn't clean. It sounds like you are saying "help me with my 
> unclean shutdown; please get rid of the debugging information that is 
> trying to help me!".
>
>> I've tried, beside a lot of other things, to set 
>> skipMemoryLeakChecksOnJvmShutdown="true" in context.xml but it seems 
>> to change nothing at all.
>
> Tomcat will never kill your non-daemon thread(s) from inside the VM.
> Since you are using a CATALINA_PID environment variable (and, 
> therefore, a pid-file), the shutdown process is sending the TERM 
> signal. It's also possibly sending a KILL signal, depending on whether 
> or not the TERM worked and if -force was supplied on the command-line.

Unfortunately 

Re: Tomcat 9 doesn't shutdown cleanly

[Simon Matter]

Hi Chris,

Thank you for the quick reply.

> Simon,
>
> On 11/30/21 08:21, Simon Matter wrote:
>> I'm running an application on Tomcat 9.0.55 on x86_64 Linux with OpenJDK
>> JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.
>>
>> When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly
>> (most
>> threads are gone) but send a thread dump to catalina.out and is later
>> killer by an OS signal.
>
> This should only happen if you have CATALINA_PID defined. Are you indeed
> defining that environment variable?
>
> catalina.sh has this code in it, which is probably what you are seeing:
>
>echo "To aid diagnostics a thread dump has been written to
> standard out."
>kill -3 `cat "$CATALINA_PID"`
>

That's true, I have CATALINA_PID defined when the call of "catalina.sh
start" is done. So yes, the script will kill the java process if it
doesn't terminate.

>> When shutting down Tomcat via the shutdown listener on port 8005, it
>> also
>> shuts down mostly but without the thread dump in catalina.out. Sending
>> SIGTERM later to the still running java process terminates it.
>
> Right: when you manually connect to the shutdown port and send
> "SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't send a
> signal. You have to do that manually, too.

On a test host, when I send "SHUTDOWN" to the shutdown listener, Tomcat
shuts down and the Java VM terminates.

Only on this host with the application, when I send "SHUTDOWN" to the
shutdown listener, Tomcat shuts down but the Java VM doesn't terminate.

>
>> So both methods somehow terminate Tomcat partly but not completely.
>
> You have one or two of the following issues:
>
> 1. You have a non-daemon thread running
> 2. You have an unusually long shutdown process that just takes "too long"
>
> The good news is that the thread-dup can answer that question: what's in
> the thread dump that is generated when you run "catalina.sh stop"?
>
>> When simply sending SIGTERM on the OS level, Tomcat shuts down cleanly
>> and
>> terminates without issues.
>>
>> One thing in common is that I always see these messages while shutting
>> down:
>>
>> 30-Nov-2021 13:59:36.985 SEVERE [main]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
>> Found RMI Target with stub class class
>> [sun.rmi.registry.RegistryImpl_Stub] and value
>> [RegistryImpl_Stub[UnicastRef [liveRef:
>> [endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
>> Target has been forcibly removed to prevent a memory leak.
>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
>> Found RMI Target with stub class class [com.sun.proxy.$Proxy51] and
>> value
>> [Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastRef
>> [liveRef:
>> [endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7ffd,
>> -4005526521234186948]]. This RMI Target has been forcibly removed to
>> prevent a memory leak.
>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
>> Found RMI Target with stub class class
>> [sun.rmi.registry.RegistryImpl_Stub] and value
>> [RegistryImpl_Stub[UnicastRef [liveRef:
>> [endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI
>> Target has been forcibly removed to prevent a memory leak.
>>
>> Why do the three methods to shutdown Tomcat differ and how can I make
>> 'catalina.sh stop' or the shutdown listener work the same way like
>> sending
>> the OS signal.
>
> What's the difference? You don't want the thread dump in your
> catalina.out? That's supposed to be helpful in debugging why your
> shutdown isn't clean. It sounds like you are saying "help me with my
> unclean shutdown; please get rid of the debugging information that is
> trying to help me!".
>
>> I've tried, beside a lot of other things, to set
>> skipMemoryLeakChecksOnJvmShutdown="true" in context.xml but it seems to
>> change nothing at all.
>
> Tomcat will never kill your non-daemon thread(s) from inside the VM.
> Since you are using a CATALINA_PID environment variable (and, therefore,
> a pid-file), the shutdown process is sending the TERM signal. It's also
> possibly sending a KILL signal, depending on whether or not the TERM
> worked and if -force was supplied on the command-line.

Unfortunately I am not an insider when it comes to Java, so I'm not sure
what a non-daemon thread means here? Can these be threads started by our
application which are non-daemon threads? So they will be terminated with
SIGTERM to the Java VM but not using the shutdown listener?

I'll paste a thread dump below.

Thanks,
Simon

Full thread dump OpenJDK 64-Bit Server VM (11.0.13+8 mixed mode):

Threads class SMR info:
_java_thread_list=0x55a49ac4ba60, length=18, elements={
0x7f1ac0148800, 0x7f1ac014a800, 0x7f1ac015f800,
0x7f1ac0161800,
0x7f1ac0164000, 0x7f1ac0168800, 0x

Re: Tomcat 9 doesn't shutdown cleanly

[Christopher Schultz]

Simon,

On 11/30/21 08:21, Simon Matter wrote:

I'm running an application on Tomcat 9.0.55 on x86_64 Linux with OpenJDK
JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.

When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly (most
threads are gone) but send a thread dump to catalina.out and is later
killer by an OS signal.


This should only happen if you have CATALINA_PID defined. Are you indeed 
defining that environment variable?


catalina.sh has this code in it, which is probably what you are seeing:

  echo "To aid diagnostics a thread dump has been written to 
standard out."

  kill -3 `cat "$CATALINA_PID"`


When shutting down Tomcat via the shutdown listener on port 8005, it also
shuts down mostly but without the thread dump in catalina.out. Sending
SIGTERM later to the still running java process terminates it.


Right: when you manually connect to the shutdown port and send 
"SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't send a 
signal. You have to do that manually, too.



So both methods somehow terminate Tomcat partly but not completely.


You have one or two of the following issues:

1. You have a non-daemon thread running
2. You have an unusually long shutdown process that just takes "too long"

The good news is that the thread-dup can answer that question: what's in 
the thread dump that is generated when you run "catalina.sh stop"?



When simply sending SIGTERM on the OS level, Tomcat shuts down cleanly and
terminates without issues.

One thing in common is that I always see these messages while shutting down:

30-Nov-2021 13:59:36.985 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class [com.sun.proxy.$Proxy51] and value
[Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastRef
[liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7ffd,
-4005526521234186948]]. This RMI Target has been forcibly removed to
prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.

Why do the three methods to shutdown Tomcat differ and how can I make
'catalina.sh stop' or the shutdown listener work the same way like sending
the OS signal.


What's the difference? You don't want the thread dump in your 
catalina.out? That's supposed to be helpful in debugging why your 
shutdown isn't clean. It sounds like you are saying "help me with my 
unclean shutdown; please get rid of the debugging information that is 
trying to help me!".



I've tried, beside a lot of other things, to set
skipMemoryLeakChecksOnJvmShutdown="true" in context.xml but it seems to
change nothing at all.


Tomcat will never kill your non-daemon thread(s) from inside the VM. 
Since you are using a CATALINA_PID environment variable (and, therefore, 
a pid-file), the shutdown process is sending the TERM signal. It's also 
possibly sending a KILL signal, depending on whether or not the TERM 
worked and if -force was supplied on the command-line.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 9 doesn't shutdown cleanly

[Simon Matter]

Hi,

I'm running an application on Tomcat 9.0.55 on x86_64 Linux with OpenJDK
JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.

When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly (most
threads are gone) but send a thread dump to catalina.out and is later
killer by an OS signal.

When shutting down Tomcat via the shutdown listener on port 8005, it also
shuts down mostly but without the thread dump in catalina.out. Sending
SIGTERM later to the still running java process terminates it.

So both methods somehow terminate Tomcat partly but not completely.

When simply sending SIGTERM on the OS level, Tomcat shuts down cleanly and
terminates without issues.

One thing in common is that I always see these messages while shutting down:

30-Nov-2021 13:59:36.985 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class [com.sun.proxy.$Proxy51] and value
[Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastRef
[liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7ffd,
-4005526521234186948]]. This RMI Target has been forcibly removed to
prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.

Why do the three methods to shutdown Tomcat differ and how can I make
'catalina.sh stop' or the shutdown listener work the same way like sending
the OS signal.

I've tried, beside a lot of other things, to set
skipMemoryLeakChecksOnJvmShutdown="true" in context.xml but it seems to
change nothing at all.

Any help would be much appreciated.

Thanks,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org