We pulled a host from the pool for upgrades, and in the process seems to
have gotten a virtual router in an odd state. It's showing as destroyed in
the UI, but cloudmonkey says it's still expunging.
This host has been completely rebuild including completely redisked. On the
management node I found:
mysql> select * from op_ha_work
-> ;
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+
| id | instance_id | type | vm_type | state | mgmt_server_id
| host_id | created | tried | taken | step | time_to_try |
updated |
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+
| 1 | 13 | Migration | DomainRouter | Expunging | NULL
| 24 | 2016-07-01 14:34:17 | 0 | NULL | Migrating | 1433332034 |
205 |
| 4 | 78 | Migration | DomainRouter | Destroyed | NULL
| 24 | 2016-07-01 14:34:17 | 0 | NULL | Migrating | 1433332092 |
68 |
+----+-------------+-----------+--------------+-----------+----------------+---------+---------------------+-------+-------+-----------+-------------+---------+
I removed those entries, but when the hosts persist. Via cloudmonkey it
shows expunging:
{
"count": 1,
"router": [
{
"account": "engineering",
"created": "2014-09-05T03:56:07+0200",
"dns1": "172.16.8.46",
"dns2": "172.16.8.47",
"domain": "engineering",
"domainid": "1da498ba-5646-4cc3-a704-a20ebe12f518",
"id": "dc48a402-41d8-4e93-b441-4b34eb83a4c8",
"isredundantrouter": true,
"name": "r-78-VM",
"nic": [],
"podid": "f53afa8d-51ff-484d-9a88-52e979aeb688",
"redundantstate": "UNKNOWN",
"requiresupgrade": false,
"role": "VIRTUAL_ROUTER",
"serviceofferingid": "ed6b13d0-3e74-4aa5-a6b7-a5d2ac6c4a6c",
"serviceofferingname": "System Offering For Software Router",
"state": "Expunging",
"templateid": "bb3f7e4e-d7f6-4a72-a752-12c3221e43e9",
"version": "4.4.1",
"zoneid": "3467ff63-b582-4ace-9fda-8d5851bd8753",
"zonename": "Oakland"
}
]
}
If I try to destroy the host from the api I get:
Async job cf08d7fa-1609-4d0e-b33c-63cc38f7e897 failed
Error 530, Unable to locate datastore with id 18
{
"accountid": "e3389462-6020-425a-9b9e-57141d58e1ab",
"cmd": "org.apache.cloudstack.api.command.admin.router.DestroyRouterCmd",
"created": "2016-07-05T17:23:53+0200",
"jobid": "cf08d7fa-1609-4d0e-b33c-63cc38f7e897",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Unable to locate datastore with id 18"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "xxx"
}
I'm guessing I need to remove all references for the routers from the
database. Does anyone know what table(s) that's stored in?
