Hi Robert
Thanks for your suggestion! Currently our postgres max_connection is 100 and the VDB connection pool limit is 50. Usually concurrent connections should not exceed these numbers, but I will try raise them and watch the postgres instances Thanks! Francis _____ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Robert Hodges Sent: Friday, December 21, 2007 10:31 AM To: Sequoia general mailing list Subject: Re: [Sequoia] Error during backup: (No more connections) Hi Francis, I think this might be happening because load transfers over to the other controller when you take a backend off-line for backup. This raises the number of connections on that controller's backend. Can you check your value for max_connections in postgresql.conf? We typically see error messages when PostgreSQL decides to stop handing out connections. Run 'ps -ef |grep postgres' on the PostgreSQL host to see how close you are sailing to the wind when your application is running. You can cure it by raising the value and rebooting PostgreSQL. There's no harm to increasing it so long as your machine is not overloaded. Another cure is to limit the number of connections you allow in the connection pool for your VDB(s). Cheers, Robert Robert Hodges, CTO, Continuent, Inc. Email: [EMAIL PROTECTED] Mobile: +1-510-501-3728 Skype: hodgesrm On Dec 20, 2007, at 6:09 PM, Francis Chong wrote: Hi, We have just encountered some wired problem during backup Sequoia backend, don't sure if this is a bug. Setup: Two controllers at two servers, each connecting to one Postgres database backend. Client using "ordered" load balancing to connect to controller 1 then controller 2. Background: After a (simulated) busy day, I tried to backup my db (in CLI, enter command like backup). It is fine when I backup controller 1/backend 1. When I backup controller 2/backend 2 however, I encounter some problem that lead the controller stop responding. Problem: 1. Issued backup command. The backend is disabled and controller run pg_dump to backup 2. Request failed with cause " (No more connections)". We should have the backend disabled but its still having incoming request? 3. The backend is disabled due to the error. When I restore using another controller's healthy backup, it stops responding. Any idea? I have attached related logs below: 2007-12-21 09:18:35,203 INFO controller.RequestManager.s2 All activity is suspendedfor s2 2007-12-21 09:18:35,208 INFO controller.virtualdatabase.s2 Checkpoint backup middleare1-20071221-0918-128.128.3.27:25322-20071221091835197+0800 was stored 2007-12-21 09:18:35,210 INFO controller.virtualdatabase.s2 Disabling backend middleware1, waiting for current transaction to complete on this backend 2007-12-21 09:18:35,211 INFO controller.RequestManager.s2 Setting new virtual database schema. 2007-12-21 09:18:35,214 INFO sequoia.controller.connection 5 connections freed on "jdbc:postgresql://128.128.3.27:5432/s" 2007-12-21 09:18:35,377 INFO controller.virtualdatabase.s2 Backend middleware1 is now disabled 2007-12-21 09:18:35,377 INFO controller.virtualdatabase.s2 Backend middleware1 disabled on controller Member(address=/128.128.3.27:34667, uid=128.128.3.27:34667) 2007-12-21 09:18:35,390 INFO controller.RequestManager.s2 All activity is now resumed for s2 2007-12-21 09:18:35,391 INFO controller.RequestManager.s2 Backend middleware1 is now disabled 2007-12-21 09:18:35,391 INFO controller.RequestManager.s2 Starting backup of backend middleware1 2007-12-21 09:18:35,430 INFO backup.backupers.NativeCommandExec Starting execution of "pg_dump -h 128.128.3.27 -p 5432 -U postgres --format=c -f /tmp/backup/middleware1-20071221-0918 s" 2007-12-21 09:19:19,664 WARN controller.RequestManager.s2 Request 844424932469376 failed on controller Member(address=/128.128.3.30:45147, uid=128.128.3.30:45147) (java.sql.SQLException: stored procedure 844424932469376 failed: Backend s2 - BackendWorkerThread for backend 'middleware2' with RAIDb level:1 failed(No more connections)) 2007-12-21 09:19:19,706 WARN controller.RequestManager.s2 Request {call nextval(?)}/<!%S|CFM|!%> failed (stored procedure 844424932469376 failed:Backend s2 - BackendWorkerThread for backend 'middleware2' with RAIDb level:1 failed(No more connections)) 2007-12-21 09:19:19,706 WARN controller.virtualdatabase.s2 Stored procedure 844424932469376 failed (stored procedure 844424932469376 failed:Backend s2 - BackendWorkerThread for backend 'middleware2' with RAIDb level:1 failed(No more connections)) 2007-12-21 09:19:19,707 WARN virtualdatabase.VirtualDatabaseWorkerThread.s2 Error during command execution (stored procedure 844424932469376 failed: Backend s2 - BackendWorkerThread for backend 'middleware2' with RAIDb level:1 failed (No more connections) 2007-12-21 09:20:00,239 INFO backup.backupers.NativeCommandExec Command "pg_dump -h 128.128.3.27 -p 5432 -U postgres --format=c -f /tmp/backup/middleware1-20071221-0918 s" logged 0 errors and terminated with exitcode 0 2007-12-21 09:20:00,242 INFO controller.RequestManager.s2 Backup has completed. 2007-12-21 09:20:00,417 INFO controller.recoverylog.RecoverThread Starting recovery 2007-12-21 09:20:04,301 INFO controller.RequestManager.s2 All activity is suspended for s2 2007-12-21 09:20:04,303 INFO controller.recoverylog.RecoverThread Recovery completed 2007-12-21 09:21:04,306 WARN sequoia.controller.scheduler Timeout in begin, still waiting for 0 pending transactions to completec 2007-12-21 09:21:04,306 WARN controller.virtualdatabase.s2 Begin failed (Timeout in begin, still waiting for 0 pending transactions to complete) 2007-12-21 09:21:04,306 WARN virtualdatabase.VirtualDatabaseWorkerThread.s2 Error during command execution (Timeout in begin, still waiting for 0 pending transactions to complete) 2007-12-21 09:21:19,703 WARN sequoia.controller.scheduler Timeout in begin, still waiting for 0 pending transactions to complete Regards Francis _______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
_______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
