[ https://forge.continuent.org/jira/browse/SEQUOIA-956?page=comments#action_13990 ]
xavier roques commented on SEQUOIA-956: --------------------------------------- My understanding is that the RAIDb1.disableBackend should call, at the beginning, backend.setState (BackendState.DISABLING) and not db.disable(). The method db.disable() should only be called at the end, when the database is really disabled. > Process hang when a backend is disabled > --------------------------------------- > > Key: SEQUOIA-956 > URL: https://forge.continuent.org/jira/browse/SEQUOIA-956 > Project: Sequoia > Type: Bug > Reporter: xavier roques > > > I have a VirtualDatabase with two backends. > My loadbalancer is a RAIDb1_WRR load balancer. > During an heavy database process, I cut the network with one of my databases. > The databases is disabled OK. > But my process hang > I checked the threads call stacks and I can see all the > VirtualDatabaseWorkerThread were blocked. > Below is the stack: > INFO | jvm 1 | 2007/07/06 15:32:42 | java.lang.Thread.State: BLOCKED > (on object monitor) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.backend.DatabaseBackend.isReadEnabled(DatabaseBackend.java:1497) > INFO | jvm 1 | 2007/07/06 15:32:42 | - waiting to lock > <0x082fdba8> (a org.continuent.sequoia.controller.backend.DatabaseBackend) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1_WRR.executeWRR(RAIDb1_WRR.java:195) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1_WRR.execSingleBackendReadRequest(RAIDb1_WRR.java:112) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1.statementExecuteQuery(RAIDb1.java:160) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.requestmanager.RequestManager.statementExecuteQuery(RequestManager.java:607) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.statementExecuteQuery(VirtualDatabase.java:549) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.statementExecuteQuery(VirtualDatabaseWorkerThread.java:2175) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.virtualdatabase.VirtualDatabaseWorkerThread.run(VirtualDatabaseWorkerThread.java:442) > Furthermore, I find another interesting stack: > INFO | jvm 1 | 2007/07/06 15:32:42 | java.lang.Thread.State: RUNNABLE > INFO | jvm 1 | 2007/07/06 15:32:42 | at > java.net.SocketInputStream.socketRead0(Native Method) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > java.net.SocketInputStream.read(SocketInputStream.java:129) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > java.io.BufferedInputStream.read(BufferedInputStream.java:237) > INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x08b0fec0> (a > java.io.BufferedInputStream) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > java.io.DataInputStream.readInt(DataInputStream.java:370) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.common.protocol.TypeTag.<init>(TypeTag.java:169) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.driver.Connection.receiveBooleanOrException(Connection.java:2176) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.driver.Connection.receiveSQLWarnings(Connection.java:2112) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.driver.Connection.statementExecuteQuery(Connection.java:2763) > INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x08afd4f0> (a > org.continuent.sequoia.driver.Connection) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.driver.Statement.executeQuery(Statement.java:522) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.driver.PreparedStatement.executeQuery(PreparedStatement.java:168) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.accesslayer.JdbcAccessImpl.executeQuery(JdbcAccessImpl.java:291) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.accesslayer.RsQueryObject.performQuery(RsQueryObject.java:72) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.accesslayer.RsIterator.<init>(RsIterator.java:187) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.core.RsIteratorFactoryImpl.createRsIterator(RsIteratorFactoryImpl.java:58) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.core.PersistenceBrokerImpl.getRsIteratorFromQuery(PersistenceBrokerImpl.java:2109) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.core.PersistenceBrokerImpl.getIteratorFromQuery(PersistenceBrokerImpl.java:1664) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.core.PersistenceBrokerImpl.getObjectByQuery(PersistenceBrokerImpl.java:1790) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.core.DelegatingPersistenceBroker.getObjectByQuery(DelegatingPersistenceBroker.java:352) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.apache.ojb.broker.core.DelegatingPersistenceBroker.getObjectByQuery(DelegatingPersistenceBroker.java:352) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > com.infovista.mserver.DbManager.getValidatedSlave(DbManager.java:1862) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > com.infovista.mserver.JMXManager.backupDisabled(JMXManager.java:1388) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > com.infovista.mserver.actions.CJDBCLaunch$2.handleNotification(CJDBCLaunch.java:476) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.backend.DatabaseBackend.sendNotification(DatabaseBackend.java:1643) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.backend.DatabaseBackend.notifyJMXStateChanged(DatabaseBackend.java:1578) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.backend.DatabaseBackend.setState(DatabaseBackend.java:1736) > INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x082fdba8> (a > org.continuent.sequoia.controller.backend.DatabaseBackend) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.backend.DatabaseBackend.disable(DatabaseBackend.java:1260) > INFO | jvm 1 | 2007/07/06 15:32:42 | - locked <0x082fdba8> (a > org.continuent.sequoia.controller.backend.DatabaseBackend) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.raidb1.RAIDb1.disableBackend(RAIDb1.java:1992) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.tasks.StatementExecuteUpdateTask.executeInTransaction(StatementExecuteUpdateTask.java:295) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.tasks.StatementExecuteUpdateTask.executeTask(StatementExecuteUpdateTask.java:118) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.tasks.AbstractTask.execute(AbstractTask.java:140) > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.loadbalancer.BackendWorkerThread.run(BackendWorkerThread.java:197) > This last stack shows that Sequoia is trying to disable the fallen database. > My application is listening on the state of the backend. > When My application is informed, thanks to a JMX notification, that the > database is down, it tries to make a db request. > Unfortunately, this request will never finished because all the > DatabaseWorker threads are blocked in > INFO | jvm 1 | 2007/07/06 15:32:42 | at > org.continuent.sequoia.controller.backend.DatabaseBackend.isReadEnabled(DatabaseBackend.java:1497) > INFO | jvm 1 | 2007/07/06 15:32:42 | - waiting to lock > <0x082fdba8> (a org.continuent.sequoia.controller.backend.DatabaseBackend) > Last point, the configuration of my virtual database is poolThreads="false" -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://forge.continuent.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira _______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
