Hello Simon,
if you use the Registry to bind Objects / Stubs, you must also call "unbind" on
shutdown:
https://docs.oracle.com/javase/7/docs/api/java/rmi/registry/Registry.html
I think the developer who implemented the RMI stub, should also now what to
unbind.
Greetings,
Thomas
-Ursprüngliche Nachricht-
Von: Simon Matter
Gesendet: Dienstag, 30. November 2021 16:10
An: Tomcat Users List
Betreff: Re: AW: Tomcat 9 doesn't shutdown cleanly
Hi Thomas,
> Hello,
>
> it looks like your application opens several ports for RMI communication.
> One class is mentioned in your first mail: ShopdbCacheSynchronizer
Yes, and it's true that when I'm looking at the shut down Tomcat instance VM, I
see several RMI threads lingering around.
>
> Maybe you can ask the developer guys to check whether these threads /
> ports are terminated / closed cleanly on shutdown event.
> Quite often developers don't care much about shutting down their stuff
> cleanly.
Good guess, that's exactly what I'm trying to do, finding out why we have the
shutdown problems so the "developer guys" can finally fix this issue.
From how I understand it, we have an internal application server which
initiates RMI connections to the Tomcat instance sitting in the DMZ. My
question is, can we terminate the RMI connections on the Tomcat instance only
and shut down Tomcat, or do we have to close the RMI connections on the
internal appserver which initiated the connections?
Apart from RMI, is there anything in the thread dump which indicates an issue
in out Tomcat app?
Kind regards,
Simon
>
> Greetings,
> Thomas
>
> -Ursprüngliche Nachricht-
> Von: Simon Matter
> Gesendet: Dienstag, 30. November 2021 15:40
> An: Tomcat Users List
> Betreff: Re: Tomcat 9 doesn't shutdown cleanly
>
> Hi Chris,
>
> Thank you for the quick reply.
>
>> Simon,
>>
>> On 11/30/21 08:21, Simon Matter wrote:
>>> I'm running an application on Tomcat 9.0.55 on x86_64 Linux with
>>> OpenJDK
>>> JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.
>>>
>>> When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly
>>> (most threads are gone) but send a thread dump to catalina.out and
>>> is later killer by an OS signal.
>>
>> This should only happen if you have CATALINA_PID defined. Are you
>> indeed defining that environment variable?
>>
>> catalina.sh has this code in it, which is probably what you are seeing:
>>
>>echo "To aid diagnostics a thread dump has been written to
>> standard out."
>>kill -3 `cat "$CATALINA_PID"`
>>
>
> That's true, I have CATALINA_PID defined when the call of "catalina.sh
> start" is done. So yes, the script will kill the java process if it
> doesn't terminate.
>
>>> When shutting down Tomcat via the shutdown listener on port 8005, it
>>> also shuts down mostly but without the thread dump in catalina.out.
>>> Sending SIGTERM later to the still running java process terminates
>>> it.
>>
>> Right: when you manually connect to the shutdown port and send
>> "SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't
>> send a signal. You have to do that manually, too.
>
> On a test host, when I send "SHUTDOWN" to the shutdown listener,
> Tomcat shuts down and the Java VM terminates.
>
> Only on this host with the application, when I send "SHUTDOWN" to the
> shutdown listener, Tomcat shuts down but the Java VM doesn't terminate.
>
>>
>>> So both methods somehow terminate Tomcat partly but not completely.
>>
>> You have one or two of the following issues:
>>
>> 1. You have a non-daemon thread running 2. You have an unusually long
>> shutdown process that just takes "too long"
>>
>> The good news is that the thread-dup can answer that question: what's
>> in the thread dump that is generated when you run "catalina.sh stop"?
>>
>>> When simply sending SIGTERM on the OS level, Tomcat shuts down
>>> cleanly and terminates without issues.
>>>
>>> One thing in common is that I always see these messages while
>>> shutting
>>> down:
>>>
>>> 30-Nov-2021 13:59:36.985 SEVERE [main]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiT
>>> a rgets Found RMI Target with stub class class
>>> [sun.rmi.registry.RegistryImpl_Stub] and value
>>> [RegistryImpl_Stub[UnicastRef [liveRef:
>>> [endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
>>> Target has been forcibly removed to prevent a memory leak.
>>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiT
>>> a rgets Found RMI Target with stub class class
>>> [com.sun.proxy.$Proxy51] and value
>>> [Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[Unicast
>>> R
>>> ef
>>> [liveRef:
>>> [endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-
>>> 7 ffd, -4005526521234186948]]. This RMI Target has been forcibly
>>> removed to prevent a memory leak.
>>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>>>
