Process hang when a backend is disabled
---------------------------------------
Key: SEQUOIA-956
URL: https://forge.continuent.org/jira/browse/SEQUOIA-956
Project: Sequoia
Type: Bug
Reporter: xavier roques
I have a VirtualDatabase with two backends.
My loadbalancer is a RAIDb1_WRR load balancer.
During an heavy database process, I cut the network with one of my databases.
The databases is disabled OK.
But my process hang
I checked the threads call stacks and I can see all the
VirtualDatabaseWorkerThread were blocked.
Below is the stack:
INFO | jvm 1 | 2007/07/06 15:32:42 | java.lang.Thread.State: BLOCKED
(on object monitor)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.backend.DatabaseBackend.isReadEnabled(DatabaseBackend.java:1497)
INFO | jvm 1 | 2007/07/06 15:32:42 | - waiting to lock <0x082fdba8>
(a org.continuent.sequoia.controller.backend.DatabaseBackend)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1_WRR.executeWRR(RAIDb1_WRR.java:195)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1_WRR.execSingleBackendReadRequest(RAIDb1_WRR.java:112)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1.statementExecuteQuery(RAIDb1.java:160)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.requestmanager.RequestManager.statementExecuteQuery(RequestManager.java:607)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(VirtualDatabase.java:549)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(VirtualDatabaseWorkerThread.java:2175)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(VirtualDatabaseWorkerThread.java:442)
Furthermore, I find another interesting stack:
INFO | jvm 1 | 2007/07/06 15:32:42 | java.lang.Thread.State: RUNNABLE
INFO | jvm 1 | 2007/07/06 15:32:42 | at
java.net.SocketInputStream.socketRead0(Native Method)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
java.net.SocketInputStream.read(SocketInputStream.java:129)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x08b0fec0> (a
java.io.BufferedInputStream)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
java.io.DataInputStream.readInt(DataInputStream.java:370)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.common.protocol.TypeTag.<init>(TypeTag.java:169)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.driver.Connection.receiveBooleanOrException(Connection.java:2176)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.driver.Connection.receiveSQLWarnings(Connection.java:2112)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.driver.Connection.statementExecuteQuery(Connection.java:2763)
INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x08afd4f0> (a
org.continuent.sequoia.driver.Connection)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.driver.Statement.executeQuery(Statement.java:522)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.driver.PreparedStatement.executeQuery(PreparedStatement.java:168)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.accesslayer.JdbcAccessImpl.executeQuery(JdbcAccessImpl.java:291)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.accesslayer.RsQueryObject.performQuery(RsQueryObject.java:72)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.accesslayer.RsIterator.<init>(RsIterator.java:187)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.core.RsIteratorFactoryImpl.createRsIterator(RsIteratorFactoryImpl.java:58)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.core.PersistenceBrokerImpl.getRsIteratorFromQuery(PersistenceBrokerImpl.java:2109)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.core.PersistenceBrokerImpl.getIteratorFromQuery(PersistenceBrokerImpl.java:1664)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.core.PersistenceBrokerImpl.getObjectByQuery(PersistenceBrokerImpl.java:1790)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.core.DelegatingPersistenceBroker.getObjectByQuery(DelegatingPersistenceBroker.java:352)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.apache.ojb.broker.core.DelegatingPersistenceBroker.getObjectByQuery(DelegatingPersistenceBroker.java:352)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
com.infovista.mserver.DbManager.getValidatedSlave(DbManager.java:1862)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
com.infovista.mserver.JMXManager.backupDisabled(JMXManager.java:1388)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
com.infovista.mserver.actions.CJDBCLaunch$2.handleNotification(CJDBCLaunch.java:476)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.backend.DatabaseBackend.sendNotification(DatabaseBackend.java:1643)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.backend.DatabaseBackend.notifyJMXStateChanged(DatabaseBackend.java:1578)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.backend.DatabaseBackend.setState(DatabaseBackend.java:1736)
INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x082fdba8> (a
org.continuent.sequoia.controller.backend.DatabaseBackend)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.backend.DatabaseBackend.disable(DatabaseBackend.java:1260)
INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x082fdba8> (a
org.continuent.sequoia.controller.backend.DatabaseBackend)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1.disableBackend(RAIDb1.java:1992)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.tasks.StatementExecuteUpdateTask.executeInTransaction(StatementExecuteUpdateTask.java:295)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.tasks.StatementExecuteUpdateTask.executeTask(StatementExecuteUpdateTask.java:118)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.tasks.AbstractTask.execute(AbstractTask.java:140)
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.loadbalancer.BackendWorkerThread.run(BackendWorkerThread.java:197)
This last stack shows that Sequoia is trying to disable the fallen database.
My application is listening on the state of the backend.
When My application is informed, thanks to a JMX notification, that the
database is down, it tries to make a db request.
Unfortunately, this request will never finished because all the DatabaseWorker
threads are blocked in
INFO | jvm 1 | 2007/07/06 15:32:42 | at
org.continuent.sequoia.controller.backend.DatabaseBackend.isReadEnabled(DatabaseBackend.java:1497)
INFO | jvm 1 | 2007/07/06 15:32:42 | - waiting to lock <0x082fdba8>
(a org.continuent.sequoia.controller.backend.DatabaseBackend)
Last point, the configuration of my virtual database is poolThreads="false"
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://forge.continuent.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
