Hi Francis,
I think this might be happening because load transfers over to the
other controller when you take a backend off-line for backup. This
raises the number of connections on that controller's backend.
Can you check your value for max_connections in postgresql.conf? We
typically see error messages when PostgreSQL decides to stop handing
out connections. Run 'ps -ef |grep postgres' on the PostgreSQL host
to see how close you are sailing to the wind when your application is
running. You can cure it by raising the value and rebooting
PostgreSQL. There's no harm to increasing it so long as your machine
is not overloaded.
Another cure is to limit the number of connections you allow in the
connection pool for your VDB(s).
Cheers, Robert
Robert Hodges, CTO, Continuent, Inc.
Email: [EMAIL PROTECTED]
Mobile: +1-510-501-3728 Skype: hodgesrm
On Dec 20, 2007, at 6:09 PM, Francis Chong wrote:
Hi,
We have just encountered some wired problem during backup Sequoia
backend,
don't sure if this is a bug.
Setup:
Two controllers at two servers, each connecting to one Postgres
database backend. Client using "ordered" load balancing to connect to
controller 1 then controller 2.
Background:
After a (simulated) busy day, I tried to backup my db (in CLI,
enter command
like backup). It is fine when I backup controller 1/backend 1. When
I backup
controller 2/backend 2 however, I encounter some problem that lead the
controller stop responding.
Problem:
1. Issued backup command. The backend is disabled and controller
run pg_dump
to backup
2. Request failed with cause " (No more connections)". We should
have the
backend disabled but its still having incoming request?
3. The backend is disabled due to the error. When I restore using
another
controller's healthy backup, it stops responding.
Any idea? I have attached related logs below:
2007-12-21 09:18:35,203 INFO controller.RequestManager.s2 All
activity is
suspendedfor s2
2007-12-21 09:18:35,208 INFO controller.virtualdatabase.s2 Checkpoint
backup middleare1-20071221-0918-128.128.3.27:25322-20071221091835197
+0800
was stored
2007-12-21 09:18:35,210 INFO controller.virtualdatabase.s2 Disabling
backend middleware1, waiting for current transaction to complete on
this
backend
2007-12-21 09:18:35,211 INFO controller.RequestManager.s2 Setting new
virtual database schema.
2007-12-21 09:18:35,214 INFO sequoia.controller.connection 5
connections
freed on "jdbc:postgresql://128.128.3.27:5432/s"
2007-12-21 09:18:35,377 INFO controller.virtualdatabase.s2 Backend
middleware1 is now disabled
2007-12-21 09:18:35,377 INFO controller.virtualdatabase.s2 Backend
middleware1 disabled on controller Member(address=/128.128.3.27:34667,
uid=128.128.3.27:34667)
2007-12-21 09:18:35,390 INFO controller.RequestManager.s2 All
activity is
now resumed for s2
2007-12-21 09:18:35,391 INFO controller.RequestManager.s2 Backend
middleware1 is now disabled
2007-12-21 09:18:35,391 INFO controller.RequestManager.s2 Starting
backup
of backend middleware1
2007-12-21 09:18:35,430 INFO backup.backupers.NativeCommandExec
Starting
execution of "pg_dump -h 128.128.3.27 -p 5432 -U postgres --
format=c -f
/tmp/backup/middleware1-20071221-0918 s"
2007-12-21 09:19:19,664 WARN controller.RequestManager.s2 Request
844424932469376 failed on controller Member(address=/
128.128.3.30:45147,
uid=128.128.3.30:45147) (java.sql.SQLException: stored procedure
844424932469376 failed: Backend s2 - BackendWorkerThread for backend
'middleware2' with RAIDb level:1 failed(No more connections))
2007-12-21 09:19:19,706 WARN controller.RequestManager.s2 Request
{call
nextval(?)}/<!%S|CFM|!%> failed (stored procedure 844424932469376
failed:Backend s2 - BackendWorkerThread for backend 'middleware2'
with RAIDb
level:1 failed(No more connections))
2007-12-21 09:19:19,706 WARN controller.virtualdatabase.s2 Stored
procedure
844424932469376 failed (stored procedure 844424932469376
failed:Backend s2 -
BackendWorkerThread for backend 'middleware2' with RAIDb level:1
failed(No
more connections))
2007-12-21 09:19:19,707 WARN
virtualdatabase.VirtualDatabaseWorkerThread.s2
Error during command execution (stored procedure 844424932469376
failed:
Backend s2 - BackendWorkerThread for backend 'middleware2' with RAIDb
level:1 failed (No more connections)
2007-12-21 09:20:00,239 INFO backup.backupers.NativeCommandExec
Command
"pg_dump -h 128.128.3.27 -p 5432 -U postgres --format=c -f
/tmp/backup/middleware1-20071221-0918 s" logged 0 errors and
terminated with
exitcode 0
2007-12-21 09:20:00,242 INFO controller.RequestManager.s2 Backup has
completed.
2007-12-21 09:20:00,417 INFO controller.recoverylog.RecoverThread
Starting
recovery
2007-12-21 09:20:04,301 INFO controller.RequestManager.s2 All
activity is
suspended for s2
2007-12-21 09:20:04,303 INFO controller.recoverylog.RecoverThread
Recovery
completed
2007-12-21 09:21:04,306 WARN sequoia.controller.scheduler Timeout
in begin,
still waiting for 0 pending transactions to completec
2007-12-21 09:21:04,306 WARN controller.virtualdatabase.s2 Begin
failed
(Timeout in begin, still waiting for 0 pending transactions to
complete)
2007-12-21 09:21:04,306 WARN
virtualdatabase.VirtualDatabaseWorkerThread.s2
Error during command execution (Timeout in begin, still waiting for 0
pending transactions to complete)
2007-12-21 09:21:19,703 WARN sequoia.controller.scheduler Timeout
in begin,
still waiting for 0 pending transactions to complete
Regards
Francis
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
