[ https://forge.continuent.org/jira/browse/SEQUOIA-1033?page=comments#action_14267 ]
Emmanuel Cecchet commented on SEQUOIA-1033: ------------------------------------------- I don't think that this is acceptable since a disabled backend is no more in sync which means that reads might not respect a serializable order and access stale data. What is strange in the stack trace is that active connections have to be destroyed. As far as I remember, all connections had to terminate properly first before being able to backup a node. Is this some sort of a forced disable? Pending read requests could also interfere with the backup mechanism if it has to acquire exclusive locks on tables. Finally, isn't some of this code also shared with the general backend disable operation? In which case the reads could fail if the backend is stopped before the completion of the operation. As a side note on the fix, you should always prevent code like 'if (req instanceof SelecRequest)' since this has to go through the complex hierarchy of classes of AbstractRequest and it is a very slow process. AbstractRequest defines methods like isSelect() that have been designed to prevent such operation. > backup system throws exception because pending read request has not been > processed > ----------------------------------------------------------------------------------- > > Key: SEQUOIA-1033 > URL: https://forge.continuent.org/jira/browse/SEQUOIA-1033 > Project: Sequoia > Type: Bug > Components: Backup System > Versions: Sequoia 2.10.9 > Environment: RHEL4/Ubuntu 7.04 with MySQL 5.0.38 > Reporter: Robert Hodges > Assignee: Robert Hodges > Fix For: sequoia 2.10.10 > > > Backup process throws an exception if there is a pending request > (DatabaseBackend.getPendingReequests()). The process to disable before taking > a backup does not wait for pending read requests in active transactions or > persistentConnections in > DatabaseBackend.waitForAllTransactionsAndPersistentConnectionsToComplete(). > The correct fix may be to wait for all pending requests in > waitForAllTransactionsAndPersistentConnectionsToComplete() as well. > Here is a typical stack trace. > 2008-02-07 08:50:04,110 WARN sequoia.controller.connection Some connections > are still active, destroying them. > 2008-02-07 08:50:04,238 INFO controller.virtualdatabase.primUser Backend > c1_db is now disabled > 2008-02-07 08:50:04,238 INFO controller.virtualdatabase.primUser Backend > c1_db disabled on controller Member(address=c1host/10.237.152.24:6523, > uid=vdb1) > 2008-02-07 08:50:04,246 INFO controller.RequestManager.vdb1 Resuming activity > for primUser > 2008-02-07 08:50:04,255 INFO controller.RequestManager.vdb1 All activity is > now resumed for primUser > 2008-02-07 08:50:04,255 INFO controller.RequestManager.vdb1 Backend c1_db is > now disabled > 2008-02-07 08:50:04,255 INFO controller.RequestManager.vdb1 Starting backup > of backend c1_db > 2008-02-07 08:50:04,255 WARN controller.RequestManager.vdb1 Pending:SELECT > sleep(10) > 2008-02-07 08:50:04,255 WARN controller.RequestManager.vdb1 Pending Requests:1 > 2008-02-07 08:50:04,255 WARN controller.RequestManager.vdb1 Read enabled:false > 2008-02-07 08:50:04,255 WARN controller.RequestManager.vdb1 Write > enabled:false > 2008-02-07 08:50:04,256 ERROR controller.RequestManager.vdb1 Backup command > failed > org.continuent.sequoia.common.exceptions.BackupException: Backend is not > ready for backup: 1 remaining pending requests > at > org.continuent.sequoia.common.exceptions.SequoiaException.fillInStackTrace(SequoiaException.java:103) > at java.lang.Throwable.<init>(Throwable.java:196) > at java.lang.Exception.<init>(Exception.java:41) > at > org.continuent.sequoia.common.exceptions.SequoiaException.<init>(SequoiaException.java:57) > at > org.continuent.sequoia.common.exceptions.BackupException.<init>(BackupException.java:50) > at > org.continuent.sequoia.controller.requestmanager.RequestManager.backupBackend(RequestManager.java:2887) > at > org.continuent.sequoia.controller.virtualdatabase.VirtualDatabase.backupBackend(VirtualDatabase.java:2366) > at > org.continuent.sequoia.controller.virtualdatabase.management.VirtualDatabase.backupBackend(VirtualDatabase.java:403) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at > com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414) > at javax.management.StandardMBean.invoke(StandardMBean.java:323) > at > com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213) > at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784) > at > org.continuent.sequoia.controller.jmx.ChainedMBeanServer.invoke(ChainedMBeanServer.java:318) > at > org.continuent.sequoia.controller.jmx.AuthenticatingMBeanServer.invoke(AuthenticatingMBeanServer.java:127) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1410) > at > javax.management.remote.rmi.RMIConnectionImpl.access$100(RMIConnectionImpl.java:81) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1247) > at java.security.AccessController.doPrivileged(Native Method) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1350) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:784) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) > at sun.rmi.transport.Transport$1.run(Transport.java:153) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:149) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707) > at java.lang.Thread.run(Thread.java:595) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://forge.continuent.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira _______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
