Backgroud: We have a web based application running on tomcat (5.5.9) on an Gentoo (2.6.16) linux distribution. It uses hibernate (3.0), MySQL database (4.10) and MySQL J/connector(5.0.4) for database connectivity. Our application runs on a cluster of four web servers and two database servers. We use MySQL replication to maintain data in sync on the two databases, however currently none of the web servers direclty read/write from the second database servers - it is there just for failover purpose. As our database load increases, we are looking at possible solutions for loadbalancing the database requests over the two databases. For this purpose, we have been investigating Sequioa in our test environments for about a month or so. Now we have decided to slowly introduce it in our production environment by first enabling Sequioa on only one of the web servers.
Sequoia configuration. We are using collocated controller configuration in a a RAIDb-1 configuration. We have setup two controllers, one on each of the two database servers, with one database backend each. We have not enabled controller replication (therefore two controllers do not know about each other), instead we are using MySQL replication to replicate data between the two database servers. We have configured one of the web servers to use sequoia and load balance requests between the database servers 1 and 2 (using roundRobin algorithm). The other three web servers are configured to send read/write requests to just database server 1. Problems 1: I configured our production environment as described above -during low usage period. All started well, however couple of hours later, I got the errors belows. 14:02:36,901 ERROR controller.loadbalancer.RAIDb1 write request 153210 failed: Backend dbXXXX - BackendWorkerThread for backend 'dbXXXX' with RAIDb level:1 failed (Lock wait timeout exceeded; try restarting transaction) 14:02:36,906 WARN controller.virtualdatabase.dbXXXX Request update CACHE set CREATED_BY=?, CACHE_KEY... failed, transaction has been aborted (write request 153210 failed: Backend dbXXXX - BackendWorkerThread for backend 'dbXXXX' with RAIDb level:1 failed (Lock wait timeout exceeded; try restarting transaction) ) 14:02:36,907 WARN virtualdatabase.VirtualDatabaseWorkerThread.dbXXXX Error during command execution (write request 153210 failed: Backend dbXXXX - BackendWorkerThread for backend 'dbXXXX' with RAIDb level:1 failed (Lock wait timeout exceeded; try restarting transaction) ) 14:02:36,984 WARN virtualdatabase.VirtualDatabaseWorkerThread.dbXXXX Transaction 6253 was aborted by database 14:03:09,456 INFO sequoia.controller.loadbalancer Deadlock detected on backend dbXXXX, aborting transaction 6254 14:03:09,457 ERROR controller.loadbalancer.RAIDb1 write request 153428 failed: Backend dbXXXX - BackendWorkerThread for backend 'dbXXXX' with RAIDb level:1 failed (Transaction aborted due to deadlock) 14:03:09,457 WARN controller.virtualdatabase.dbXXXX Request delete from VIEW_STATE where VIEW_STATE_... failed, transaction has been aborted (write request 153428 failed: Backend dbXXXX - BackendWorkerThread for backend 'dbXXXX' with RAIDb level:1 failed (Transaction aborted due to deadlock) ) 14:03:09,457 WARN virtualdatabase.VirtualDatabaseWorkerThread.dbXXXX Error during command execution (write request 153428 failed: Backend dbXXXX - BackendWorkerThread for backend 'dbXXXX' with RAIDb level:1 failed (Transaction aborted due to deadlock) ) 14:03:09,473 WARN virtualdatabase.VirtualDatabaseWorkerThread.dbXXXX Transaction 6254 was aborted by database All our database tables are Innodb tables and our applicatoin uses transaction. We never get "lock wait timeout or deadlock" errors when our application is not running via sequioa. I know there is not much logging here but can you shed some light on the cause of these errors? Our test environment is configured very similarly to the production environment, except with much less data and user load. We have been using Sequioa in the test environment without such problems. Problems 2: About 10 minutes after problem 1 occured, the web server which was using sequoia to load balance requests "locked up " unexpectantly. We were not able to ping /telnet to it or anything. It was just not responding. Eventually we had to reboot the machine to bring it back up. I turned off sequioa and so far no problems with this machine has occured. There wsa no useful information in the logs (include sequoia logs) to indicate what happened. Do you have any thoughts here? We have not seen this machine do this before and therefore suspect that it must have somethign to do with the usage of sequoia - however no proof or idea what caused it? (note, controllers are not running on the web servers, they are runing on the database servers). *********************************************************** Squoia configuration files for one of the controller (the other has exacty the same configuration): Controller.xml: <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE SEQUOIA-CONTROLLER PUBLIC "-//Continuent//DTD SEQUOIA-CONTROLLER 2.10.2//EN" "http://sequoia.continuent.org/dtds/sequoia-controller-2.10.2.dtd";> <SEQUOIA-CONTROLLER> <Controller port="25322"> <Report hideSensitiveData="true" generateOnShutdown="true" generateOnFatal="true" enableFileLogging="true" /> <JmxSettings> <RmiJmxAdaptor port="1090"/> </JmxSettings> <VirtualDatabase configFile="dbXXXX-raidb1.xml" virtualDatabaseName="dbXXXX" autoEnableBackends="true" /> </Controller> </SEQUOIA-CONTROLLER> Virtualdatase.xml: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE SEQUOIA PUBLIC "-//Continuent//DTD SEQUOIA 2.10.2//EN" "http://sequoia.continuent.org/dtds/sequoia-2.10.2.dtd";> <SEQUOIA> <VirtualDatabase name="dbXXXX"> <AuthenticationManager> <Admin> <User username="admin" password="XXXXX"/> </Admin> <VirtualUsers> <VirtualLogin vLogin="tradeideasm" vPassword="XXXX"/> </VirtualUsers> </AuthenticationManager> <DatabaseBackend name="dbXXXX" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://XXXXX:3306/dbXXXX" connectionTestStatement="SELECT 1"> <ConnectionManager vLogin="tradeideasm" > <VariablePoolConnectionManager initPoolSize="5" minPoolSize="5" maxPoolSize="200" idleTimeout="3000" waitTimeout="10000"/> </ConnectionManager> </DatabaseBackend> <RequestManager> <RequestScheduler> <RAIDb-1Scheduler level="passThrough"/> </RequestScheduler> <LoadBalancer> <RAIDb-1> <WaitForCompletion policy="first"/> <RAIDb-1-RoundRobin/> </RAIDb-1> </LoadBalancer> </RequestManager> </VirtualDatabase> </SEQUOIA> __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
_______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
