Woah, looks like there are a bunch of threads still processing. What can I do
to clean these up? Shouldn't all the worker threads for the backends have been
shut down when I shut down the virtual DB?
Full thread dump Java HotSpot(TM) Client VM (1.5.0_15-b04 mixed mode):
"RMI ConnectionExpiration-[172.24.25.235:5558]" daemon prio=6 tid=0x0b13f288
nid=0x1478 waiting on condition [0x0b56f00
0..0x0b56fa68]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.tcp.TCPChannel$Reaper.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"RMI TCP Connection(9)-172.24.25.235" daemon prio=6 tid=0x0acb8da8 nid=0x15dc
runnable [0x0b52f000..0x0b52fae8]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <0x02a81e18> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
"DestroyJavaVM" prio=6 tid=0x0026a040 nid=0x132c waiting on condition
[0x00000000..0x0006fae8]
"RemoveIdleConnectionsThread for backend:secondary" prio=6 tid=0x0b1f7e40
nid=0x1364 in Object.wait() [0x0bf8f000..0x0b
f8fae8]
at java.lang.Object.wait(Native Method)
- waiting on <0x032ab9f8> (a
org.continuent.sequoia.controller.connection.VariablePoolConnectionManager$RemoveI
dleConnectionsThread)
at java.lang.Object.wait(Unknown Source)
at
org.continuent.sequoia.controller.connection.VariablePoolConnectionManager$RemoveIdleConnectionsThread.run(V
ariablePoolConnectionManager.java:613)
- locked <0x032ab9f8> (a
org.continuent.sequoia.controller.connection.VariablePoolConnectionManager$RemoveIdleC
onnectionsThread)
"TLCDATA - BackendWorkerThread for backend 'primary' with RAIDb level:1" prio=6
tid=0x0b134c98 nid=0x14d4 in Object.wai
t() [0x0b70f000..0x0b70fbe8]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues.getNextEntryToExecute(BackendTaskQueues.jav
a:1722)
- locked <0x032a0620> (a
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues)
at
org.continuent.sequoia.controller.loadbalancer.BackendWorkerThread.run(BackendWorkerThread.java:185)
"TLCDATA - BackendWorkerThread for backend 'primary' with RAIDb level:1" prio=6
tid=0x0b0f3498 nid=0x1480 in Object.wai
t() [0x0b6cf000..0x0b6cfc68]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues.getNextEntryToExecute(BackendTaskQueues.jav
a:1722)
- locked <0x032a0620> (a
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues)
at
org.continuent.sequoia.controller.loadbalancer.BackendWorkerThread.run(BackendWorkerThread.java:185)
"TLCDATA - BackendWorkerThread for backend 'primary' with RAIDb level:1" prio=6
tid=0x0b0f0ab0 nid=0x1594 in Object.wai
t() [0x0b64f000..0x0b64fd68]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues.getNextEntryToExecute(BackendTaskQueues.jav
a:1722)
- locked <0x032a0620> (a
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues)
at
org.continuent.sequoia.controller.loadbalancer.BackendWorkerThread.run(BackendWorkerThread.java:185)
"TLCDATA - BackendWorkerThread for backend 'primary' with RAIDb level:1" prio=6
tid=0x0b0dbb18 nid=0x80 in Object.wait(
) [0x0b60f000..0x0b60f9e8]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues.getNextCommitRollbackToExecute(BackendTaskQ
ueues.java:1796)
- locked <0x032a0620> (a
org.continuent.sequoia.controller.loadbalancer.BackendTaskQueues)
at
org.continuent.sequoia.controller.loadbalancer.BackendWorkerThread.run(BackendWorkerThread.java:182)
"RMI LeaseChecker" daemon prio=6 tid=0x0b0e0cf8 nid=0xfa8 waiting on condition
[0x0b4ef000..0x0b4efbe8]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.DGCImpl$LeaseChecker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"RMI RenewClean-[172.24.25.235:3250]" daemon prio=6 tid=0x0b0dc848 nid=0x2a0 in
Object.wait() [0x0b46f000..0x0b46fce8]
at java.lang.Object.wait(Native Method)
- waiting on <0x030156c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x030156c0> (a java.lang.ref.ReferenceQueue$Lock)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"GC Daemon" daemon prio=2 tid=0x0b0ca420 nid=0x1fc in Object.wait()
[0x0b3ef000..0x0b3ef9e8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0300e868> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(Unknown Source)
- locked <0x0300e868> (a sun.misc.GC$LatencyLock)
"RMI TCP Accept-0" daemon prio=6 tid=0x0b0c9df8 nid=0x7c8 runnable
[0x0b36f000..0x0b36fae8]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0x0300e978> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"RMI TCP Accept-5558" daemon prio=6 tid=0x0b0b4050 nid=0x670 runnable
[0x0b32f000..0x0b32fb68]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0x0300ea98> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Timer-0" daemon prio=6 tid=0x0acd8358 nid=0x16e8 in Object.wait()
[0x0b2ef000..0x0b2efbe8]
at java.lang.Object.wait(Native Method)
- waiting on <0x0300ec20> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Unknown Source)
- locked <0x0300ec20> (a java.util.TaskQueue)
at java.util.TimerThread.run(Unknown Source)
"Low Memory Detector" daemon prio=6 tid=0x00a38938 nid=0x13c0 runnable
[0x00000000..0x00000000]
"CompilerThread0" daemon prio=10 tid=0x00a37520 nid=0x1654 waiting on condition
[0x00000000..0x0abaf6c8]
"Signal Dispatcher" daemon prio=10 tid=0x00a36918 nid=0x1384 waiting on
condition [0x00000000..0x00000000]
"Finalizer" daemon prio=8 tid=0x00a2d8c8 nid=0x124c in Object.wait()
[0x0ab2f000..0x0ab2fa68]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x02f71608> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
"Reference Handler" daemon prio=10 tid=0x00a2c490 nid=0x1124 in Object.wait()
[0x0aaef000..0x0aaefae8]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Unknown Source)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
- locked <0x02f71690> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00a28250 nid=0x1678 runnable
"VM Periodic Task Thread" prio=10 tid=0x00a5b448 nid=0x1354 waiting on condition
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Emmanuel Cecchet
Sent: Tuesday, September 30, 2008 9:47 AM
To: Sequoia general mailing list
Subject: Re: [Sequoia] Can't shutdown controller programmatically
Hi Jonathan,
Could you get a thread dump to see what threads remain in the controller
VM? If you started the controller in a shell just press Ctrl+\ to get
the thread dump from the JVM.
I am not sure how the RMI registry terminates when there are still
pending JMX calls, that might be the problem.
Keep us posted with your findings,
Emmanuel
> I'm trying to shut down the controller through JMX calls and the
> application never exits.
> I start the controller with the batch file included with Sequoia. When
> we're shutting down our application, we wish to shut down the
> controller. I have a VirtualDatabaseMBean called virtualDB and a
> ControllerMBean called controller.
> The code goes something like this:
> virtualDB.shutdown(Constants.SHUTDOWN_SAFE);
> waitForVirtualDBShutdownJMXMessage();
> controller.shutdown();
> There's a bit more code in there, but that's the core.
> I get a response from the VirtualDB shutdown that it's complete and
> then it proceeds to shutdown the controller. Everything appears fine,
> but I get an exception near the end of the log output. Here's the
> sequence I get:
> 09:29:50,059 INFO controller.virtualdatabase.VDB Shutting down virtual
> database VDB in safe mode : waiting for all current transactions to
> complete
> 09:29:50,059 INFO controller.virtualdatabase.VDB Waiting for virtual
> database VDB shutdown
> 09:29:50,059 INFO controller.RequestManager.VDB Setting new virtual
> database schema.
> 09:29:50,059 INFO sequoia.controller.connection 5 connections freed on
> "jdbc:derby://susd4m3865:1527/VDB"
> 09:29:50,152 INFO controller.RequestManager.VDB Backend primary is now
> disabled
> 09:29:50,152 INFO sequoia.controller.shutdown Waiting for 0 virtual
> database worker threads to terminate.
> 09:29:50,168 INFO controller.virtualdatabase.VDB The virtual database
> VDB has been successfully shut down
> Sequoia Virtual Database has been shut down
> 09:29:50,277 INFO controller.core.Controller Shutting down controller
> 172.24.25.235:25322
> 09:29:50,277 INFO controller.core.Controller Waiting for controller
> shutdown
> 09:29:50,277 INFO sequoia.controller.shutdown Starting controller shutdown
> 09:29:50,699 INFO sequoia.controller.shutdown A report has been
> generated C:\Swisslog\DB\Sequoia\log\sequoia.report
> 09:29:50,699 INFO sequoia.controller.shutdown Shutting down Jmx Agent
> 09:29:50,699 INFO controller.core.Controller ControllerServerThread
> terminating
> Sep 30, 2008 9:29:50 AM ClientCommunicatorAdmin restart
> WARNING: Failed to restart: _java.io.IOException_: Failed to get a RMI
> stub: _javax.naming.CommunicationException_ [Root exception is
> _java.rmi.NoSuchObjectException_: no such object in table]
> 09:29:51,699 INFO controller.core.PingResponder Ping responder ended
> 09:29:51,699 INFO sequoia.controller.shutdown Waiting for controller
> thread termination.
> 09:29:51,699 INFO sequoia.controller.shutdown Controller shutdown
> completed
> 09:29:51,699 INFO controller.core.Controller Shutdown of controller
> 172.24.25.235:25322 completed
> It looks like everything is ok except for the RMI problem (I'm
> assuming that has to do with shutting down the Jmx Agent), however,
> the application doesn't exit. The controller is still sitting there.
> If I try to add in a System.exit(0) command at the end of the
> shutdown() method in the controller, I get the following (which hangs
> my application):
> 09:34:45,038 INFO controller.virtualdatabase.VDB Shutting down virtual
> database VDB in safe mode : waiting for all current transactions to
> complete
> 09:34:45,038 INFO controller.virtualdatabase.VDB Waiting for virtual
> database VDB shutdown
> 09:34:45,054 INFO controller.RequestManager.VDB Setting new virtual
> database schema.
> 09:34:45,054 INFO sequoia.controller.connection 5 connections freed on
> "jdbc:derby://susd4m3865:1527/VDB"
> 09:34:45,163 INFO controller.RequestManager.VDB Backend primary is now
> disabled
> 09:34:45,163 INFO sequoia.controller.shutdown Waiting for 0 virtual
> database worker threads to terminate.
> 09:34:45,163 INFO controller.virtualdatabase.VDB The virtual database
> VDB has been successfully shut down
> 09:34:45,288 INFO controller.core.Controller Shutting down controller
> 172.24.25.235:25322
> 09:34:45,288 INFO controller.core.Controller Waiting for controller
> shutdown
> 09:34:45,288 INFO sequoia.controller.shutdown Starting controller shutdown
> 09:34:45,616 INFO sequoia.controller.shutdown A report has been
> generated C:\Swisslog\DB\Sequoia\log\sequoia.report
> 09:34:45,616 INFO sequoia.controller.shutdown Shutting down Jmx Agent
> 09:34:45,616 INFO controller.core.Controller ControllerServerThread
> terminating
> Sep 30, 2008 9:34:45 AM ClientCommunicatorAdmin restart
> WARNING: Failed to restart: _java.io.IOException_: Failed to get a RMI
> stub: _javax.naming.CommunicationException_ [Root exception is
> _java.rmi.NoSuchObjectException_: no such object in table]
> 09:34:46,616 INFO controller.core.PingResponder Ping responder ended
> 09:34:46,616 INFO sequoia.controller.shutdown Waiting for controller
> thread termination.
> 09:34:46,616 INFO sequoia.controller.shutdown Controller shutdown
> completed
> 09:34:46,616 INFO controller.core.Controller Shutdown of controller
> 172.24.25.235:25322 completed
> _java.lang.reflect.UndeclaredThrowableException_
> at $Proxy1.shutdown(Unknown Source)
> at
> com.translogiccorp.tlcappmgr.apps.sequoia.JmxClient.orderlyShutdownController(_JmxClient.java:285_)
> at
> com.translogiccorp.tlcappmgr.apps.sequoia.JmxClient.shutdownController(_JmxClient.java:238_)
> at
> com.translogiccorp.tlcappmgrarbitrator.AppMgrArbitrator.shutDownSwisslogSequoia(_AppMgrArbitrator.java:586_)
> at
> com.translogiccorp.tlcappmgrarbitrator.AppMgrArbitrator.transitionTo(_AppMgrArbitrator.java:309_)
> at
> com.translogiccorp.tlcappmgrarbitrator.AppMgrArbitrator.update(_AppMgrArbitrator.java:1424_)
> at java.util.Observable.notifyObservers(_Observable.java:142_)
> at
> com.translogiccorp.tlcappmgr.apps.tools.ObserverVector.update(_ObserverVector.java:45_)
> at java.util.Observable.notifyObservers(_Observable.java:142_)
> at
> com.translogiccorp.tlcappmgr.apps.AppHandler.dispose(_AppHandler.java:260_)
> at
> com.translogiccorp.tlcappmgr.apps.AppHandler$ProcessHandler.handleProcess(_AppHandler.java:204_)
> at
> com.translogiccorp.tlcappmgr.apps.AppHandler$ProcessHandler.run(_AppHandler.java:187_)
> Caused by: _java.rmi.UnmarshalException_: Error unmarshaling return
> header; nested exception is:
> _java.net.SocketException_: Connection reset
> at
> sun.rmi.transport.StreamRemoteCall.executeCall(_StreamRemoteCall.java:203_)
> at sun.rmi.server.UnicastRef.invoke(_UnicastRef.java:126_)
> at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown
> Source)
> at
> javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(_RMIConnector.java:972_)
> at
> javax.management.MBeanServerInvocationHandler.invoke(_MBeanServerInvocationHandler.java:201_)
> ... 12 more
> Caused by: _java.net.SocketException_: Connection reset
> at java.net.SocketInputStream.read(_SocketInputStream.java:168_)
> at java.io.BufferedInputStream.fill(_BufferedInputStream.java:218_)
> at java.io.BufferedInputStream.read(_BufferedInputStream.java:235_)
> at java.io.DataInputStream.readByte(_DataInputStream.java:241_)
> at
> sun.rmi.transport.StreamRemoteCall.executeCall(_StreamRemoteCall.java:189_)
> ... 17 more
> How do I get the application to exit after shutting down the controller?
> Jonathan Stockho
> Software Developer
> Swisslog Healthcare Solutions
> 10825 E. 47th Avenue
> Denver, CO 80239
> Phone: 303-373-7814
> FAX: 303-373-7872
> e-mail: [EMAIL PROTECTED]
> Internet: www.swisslog.com <http://www.swisslog.com>
> ------------------------------------------------------------------------------------
>
> *About Swisslog *
> Swisslog designs solutions in logistics automation that greatly reduce
> the manual flow of materials and associated information within
> healthcare facilities.
> This message may contain legally privileged or confidential
> information and is therefore addressed to the named persons only. The
> recipient should inform the sender and delete this message, if he/she
> is not named as addressee. The sender disclaims any and all liability
> for the integrity and punctuality of this message. The sender has
> activated an automatic virus scanning, but does not guarantee the
> virus free transmission of this message.
--
Emmanuel Cecchet
FTO @ Frog Thinker
Open Source Development & Consulting
--
Web: http://www.frogthinker.org
email: [EMAIL PROTECTED]
Skype: emmanuel_cecchet
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
This message may contain legally privileged or confidential information and is
therefore addressed to the named persons only. The recipient should inform the
sender and delete this message, if he/she is not named as addressee.
The sender disclaims any and all liability for the integrity and punctuality of
this message. The sender has activated an automatic virus scanning, but does
not guarantee the virus free transmission of this message.
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
