Hi,
We have just encountered some wired problem during backup Sequoia backend,
don't sure if this is a bug.
Setup:
Two controllers at two servers, each connecting to one Postgres
database backend. Client using "ordered" load balancing to connect to
controller 1 then controller 2.
Background:
After a (simulated) busy day, I tried to backup my db (in CLI, enter command
like backup). It is fine when I backup controller 1/backend 1. When I backup
controller 2/backend 2 however, I encounter some problem that lead the
controller stop responding.
Problem:
1. Issued backup command. The backend is disabled and controller run pg_dump
to backup
2. Request failed with cause " (No more connections)". We should have the
backend disabled but its still having incoming request?
3. The backend is disabled due to the error. When I restore using another
controller's healthy backup, it stops responding.
Any idea? I have attached related logs below:
2007-12-21 09:18:35,203 INFO controller.RequestManager.s2 All activity is
suspendedfor s2
2007-12-21 09:18:35,208 INFO controller.virtualdatabase.s2 Checkpoint
backup middleare1-20071221-0918-128.128.3.27:25322-20071221091835197+0800
was stored
2007-12-21 09:18:35,210 INFO controller.virtualdatabase.s2 Disabling
backend middleware1, waiting for current transaction to complete on this
backend
2007-12-21 09:18:35,211 INFO controller.RequestManager.s2 Setting new
virtual database schema.
2007-12-21 09:18:35,214 INFO sequoia.controller.connection 5 connections
freed on "jdbc:postgresql://128.128.3.27:5432/s"
2007-12-21 09:18:35,377 INFO controller.virtualdatabase.s2 Backend
middleware1 is now disabled
2007-12-21 09:18:35,377 INFO controller.virtualdatabase.s2 Backend
middleware1 disabled on controller Member(address=/128.128.3.27:34667,
uid=128.128.3.27:34667)
2007-12-21 09:18:35,390 INFO controller.RequestManager.s2 All activity is
now resumed for s2
2007-12-21 09:18:35,391 INFO controller.RequestManager.s2 Backend
middleware1 is now disabled
2007-12-21 09:18:35,391 INFO controller.RequestManager.s2 Starting backup
of backend middleware1
2007-12-21 09:18:35,430 INFO backup.backupers.NativeCommandExec Starting
execution of "pg_dump -h 128.128.3.27 -p 5432 -U postgres --format=c -f
/tmp/backup/middleware1-20071221-0918 s"
2007-12-21 09:19:19,664 WARN controller.RequestManager.s2 Request
844424932469376 failed on controller Member(address=/128.128.3.30:45147,
uid=128.128.3.30:45147) (java.sql.SQLException: stored procedure
844424932469376 failed: Backend s2 - BackendWorkerThread for backend
'middleware2' with RAIDb level:1 failed(No more connections))
2007-12-21 09:19:19,706 WARN controller.RequestManager.s2 Request {call
nextval(?)}/<!%S|CFM|!%> failed (stored procedure 844424932469376
failed:Backend s2 - BackendWorkerThread for backend 'middleware2' with RAIDb
level:1 failed(No more connections))
2007-12-21 09:19:19,706 WARN controller.virtualdatabase.s2 Stored procedure
844424932469376 failed (stored procedure 844424932469376 failed:Backend s2 -
BackendWorkerThread for backend 'middleware2' with RAIDb level:1 failed(No
more connections))
2007-12-21 09:19:19,707 WARN virtualdatabase.VirtualDatabaseWorkerThread.s2
Error during command execution (stored procedure 844424932469376 failed:
Backend s2 - BackendWorkerThread for backend 'middleware2' with RAIDb
level:1 failed (No more connections)
2007-12-21 09:20:00,239 INFO backup.backupers.NativeCommandExec Command
"pg_dump -h 128.128.3.27 -p 5432 -U postgres --format=c -f
/tmp/backup/middleware1-20071221-0918 s" logged 0 errors and terminated with
exitcode 0
2007-12-21 09:20:00,242 INFO controller.RequestManager.s2 Backup has
completed.
2007-12-21 09:20:00,417 INFO controller.recoverylog.RecoverThread Starting
recovery
2007-12-21 09:20:04,301 INFO controller.RequestManager.s2 All activity is
suspended for s2
2007-12-21 09:20:04,303 INFO controller.recoverylog.RecoverThread Recovery
completed
2007-12-21 09:21:04,306 WARN sequoia.controller.scheduler Timeout in begin,
still waiting for 0 pending transactions to completec
2007-12-21 09:21:04,306 WARN controller.virtualdatabase.s2 Begin failed
(Timeout in begin, still waiting for 0 pending transactions to complete)
2007-12-21 09:21:04,306 WARN virtualdatabase.VirtualDatabaseWorkerThread.s2
Error during command execution (Timeout in begin, still waiting for 0
pending transactions to complete)
2007-12-21 09:21:19,703 WARN sequoia.controller.scheduler Timeout in begin,
still waiting for 0 pending transactions to complete
Regards
Francis
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
