Hi,
Once the controller is properly shutdown and restarted it never recovers
properly. I am very much surprised to see the very basic functionality of
Sequoia itself is not working.
It looks like you have a problem with your setup.
2008-05-08 17:08:55,384 INFO controller.virtualdatabase.tmdp1 Resyncing
recovery log ...
2008-05-08 17:08:55,386 INFO controller.virtualdatabase.tmdp1 Resyncing from
shutdown-10.3.40.154:25322-20080508170810106+0200
2008-05-08 17:08:55,425 INFO controller.virtualdatabase.tmdp1 Storing
checkpoint now-10.3.40.156:25322-20080508170945454+0200
2008-05-08 17:08:55,430 WARN sequoia.controller.recoverylog Checkpoint
now-10.3.40.156:25322-20080508170945454+0200 was stored
2008-05-08 17:08:55,451 INFO sequoia.controller.recoverylog 3 outdated log
entries have been removed from the recovery log
Are you sure that your clocks are properly synchronized between your
controllers?
Did you setup NTP as indicated in the documentation?
2008-05-08 17:08:55,453 INFO sequoia.controller.recoverylog Checkpoint
Member(address=/10.3.40.156:59333, uid=10.3.40.156:59333) joined group
tmdp1-10.3.40.154:25322-20080508170854741+0200 (583) will be deleted.
2008-05-08 17:08:55,453 INFO sequoia.controller.recoverylog Checkpoint
now-10.3.40.156:25322-20080508170945454+0200 (584) will be deleted.
2008-05-08 17:08:55,455 INFO sequoia.controller.recoverylog 2 out of sync
checkpoints have been removed from the recovery log
These entries are really suspicious if you did a clean shutdown.
2008-05-08 17:08:55,461 WARN sequoia.controller.recoverylog Checkpoint
now-10.3.40.156:25322-20080508170945454+0200 was stored
2008-05-08 17:09:00,032 INFO controller.virtualdatabase.tmdp1 Recovery log
re-synchronized 3601 entries successfully
2008-05-08 17:09:00,035 INFO controller.virtualdatabase.tmdp1 Resyncing
recovery log done
2008-05-08 17:09:00,103 INFO controller.virtualdatabase.tmdp1 Enabling backend
backend11 from checkpoint disable
backend11-10.3.40.154:25322-20080508170809898+0200
2008-05-08 17:09:00,419 INFO DatabaseBackend.tmdp1.backend11 Detected backend
as: MySQL
2008-05-08 17:09:00,459 INFO controller.RequestManager.tmdp1 Setting new
virtual database schema.
2008-05-08 17:09:00,482 INFO controller.recoverylog.RecoverThread Starting
recovery
2008-05-08 17:09:00,510 ERROR controller.recoverylog.RecoverThread Recovery process failed to
replay: request Autocommit StatementExecuteUpdateTask 844424930131969 (INSERT INTO
_masterstatus VALUES(?, ?, ?,
?);/<!%I|580|!%><!%I|580|!%><!%O|rO0ABXNyAA5qYXZhLnV0aWwuRGF0ZWhqgQFLWXQZAwAAeHB3CAAAARnJEyXbeA==
|!%><!%I|1695362915|!%>) (Backend tmdp1 - BackendWorkerThread for backend
'backend11' with RAIDb level:1 failed (null))
I am not sure why this request failed, you should set the recovery log
logger to DEBUG in log4j.properties to get more details about the error.
LOGS from Controller-2
-----------------------------------------------------------------------------------------------------------------------
2008-05-08 17:08:06,016 INFO controller.virtualdatabase.tmdp1 Resyncing
recovery log ...
2008-05-08 17:08:06,018 INFO controller.virtualdatabase.tmdp1 No shutdown
checkpoint found in recovery log. Clearing recovery log (dirty).
2008-05-08 17:08:06,018 INFO controller.virtualdatabase.tmdp1 Please resync
manually using 'restore log'.
2008-05-08 17:08:06,020 INFO controller.virtualdatabase.tmdp1 Resyncing
recovery log done
2008-05-08 17:08:06,033 WARN controller.virtualdatabase.tmdp1 Cannot enable
backend backend21 from a known state. Resynchronize this backend by restoring a
dump.
Ok so this shows that your log is inconsistent on controller 2 and I see
no trace of a proper initialization of that log. It looks like
controller 2 was never properly initialized.
2008-05-08 17:08:32,788 INFO controller.recoverylog.RecoverThread Database
backend backend21 is now enabled
I wonder how you were able to enable that backend without a proper
recovery log. Did you use a force enable?
These logs show a completely broken install. My advice would be to
cleanup all the recovery logs and do a fresh start using the documented
initialization procedures. If you can reproduce the issue from a clean
setup, please post a new JIRA entry with the description.
Best regards,
Emmanuel
--
Emmanuel Cecchet
FTO @ Frog Thinker
Open Source Development & Consulting
--
Web: http://www.frogthinker.org
email: [EMAIL PROTECTED]
Skype: emmanuel_cecchet
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
