First I am not sure what is causing this, I have been testing 4.3 and 4.4
multiple times and didn't notice this problem till middle of last month when
I processed all xenserver updates and then tried to go to 4.4.1. I was
getting the same error that I am unable to get status of my xenservers. I
wiped out everything but the xenservers, those I just removed from the pool
and let them reboot and added them to a new pool. In ACS it says its trying
to start the SSVM and the Console VM but they stay in the starting state.
I'm sure it is because of this error. The agent show up in ACS so I don't
know why it can't talk to the xenservers. I can ping them from the
management servers.
Last login: Wed Dec 3 00:57:04 2014 from 192.168.255.6
[root@CSTW1 ~]# grep -i -E 'exception|unable|fail|invalid|leak|warn|error'
/var/log/cloudstack/management/management-server.log
2014-12-03 00:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-a0a23e42) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:00:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-f78a59e4) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:01:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-11:ctx-2e791a18) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:01:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-7bca20e8) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:02:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-12:ctx-28d48099) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-bc593a63) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:03:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-615b171d) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:03:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-07f6ba50) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-13:ctx-c2287fb5) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:04:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-26:ctx-f7d92459) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-14:ctx-a4e62e4e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-28:ctx-01bb91e3) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-30f576ea) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:06:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-29:ctx-4d6690de) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:07:27,540 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-645639e7) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@32602693: UPDATE op_ha_work SET
op_ha_work.taken='2014-12-03 05:07:27',
op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 5
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:07:27,595 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-3:ctx-c7c94958 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:07:27,645 WARN [c.c.u.d.Merovingian2]
(HA-Worker-3:ctx-c7c94958 work-5) Was unable to find lock for the
key vm_instance22 and thread id 818653380
2014-12-03 00:07:29,425 ERROR [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-35d886d3) VM Operations failed due to
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.JDBC4PreparedStatement@61807bba: DELETE FROM async_job WHERE
async_job.id= 401
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Cannot delete or update a parent row: a foreign key constraint fails
(`cloud`.`async_job_join_map`, CONSTRAINT
`fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
`async_job` (`id`))
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:07:30,702 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-c7c94958 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:07:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-15:ctx-c7dd62d6) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:07:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-30:ctx-99d14f4c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:08:27,621 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-2:ctx-94cc502e work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:08:27,678 WARN [c.c.u.d.Merovingian2]
(HA-Worker-2:ctx-94cc502e work-2) Was unable to find lock for the
key vm_instance15 and thread id 52609881
2014-12-03 00:08:29,421 ERROR [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-ed5f2ff6) VM Operations failed due to
com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
com.mysql.jdbc.PreparedStatement@15fe9d6d: DELETE FROM async_job WHERE
async_job.id= 406
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Cannot delete or update a parent row: a foreign key constraint fails
(`cloud`.`async_job_join_map`, CONSTRAINT
`fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
`async_job` (`id`))
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:08:30,736 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-94cc502e work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:08:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-16:ctx-acc87f49) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:08:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-32:ctx-efd0036e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:09:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-1d3eecd0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:09:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-33:ctx-3271eeea) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:10:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-17:ctx-27637ed9) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:10:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-34:ctx-b1f98667) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:11:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-18:ctx-45760fd3) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:11:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-36:ctx-3367d63d) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:12:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-ee2683aa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:12:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-37:ctx-d0b02603) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:13:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-19:ctx-ce10b0fa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:13:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-38:ctx-e7e4ae6b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:14:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-20:ctx-c158dd2c) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:14:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-40:ctx-7a662bea) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:15:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-0a8f1c90) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:15:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-41:ctx-4c163515) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:16:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-51c2dfd8) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:16:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-42:ctx-c670489b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:17:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-9df53feb) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:17:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-44:ctx-a921b13b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:18:29,546 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-1:ctx-4fdb5161 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:18:29,582 WARN [c.c.u.d.Merovingian2]
(HA-Worker-1:ctx-4fdb5161 work-5) Was unable to find lock for the
key vm_instance22 and thread id 2132920636
2014-12-03 00:18:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-a54d1732) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:18:32,638 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-1:ctx-4fdb5161 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:18:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-45:ctx-5d1c0c7e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 00:19:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 00:19:30,872 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:19:30,872 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Unable to
complete AsyncJobVO {id:411, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:19:02 EST 2014}, job origin:408
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:19:30,874 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Complete async job-411,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:19:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-23:ctx-0414dcdd) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:19:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-46:ctx-fc02b226) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:20:32,282 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-821b5aca) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:20:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-48:ctx-70dbfde8) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:21:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-de7b1636) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:21:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-49:ctx-154bac45) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:22:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-5c37a417) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:22:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-50:ctx-a34d4acf) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:23:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-52:ctx-4ee4cdd0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:23:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-7:ctx-de025548) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:24:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-53:ctx-198b066a) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:24:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-27:ctx-e4b78a1b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:25:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-14:ctx-d0466cfa) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:25:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-55:ctx-6c330223) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:26:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-2:ctx-98d6b222) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:26:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-57:ctx-031552f0) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:27:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-29:ctx-a436d05f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:27:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-58:ctx-0f3496a2) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:28:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-15:ctx-40cf6a78) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:28:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-59:ctx-1537f233) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:29:30,874 ERROR [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-84af6112) Caught this throwable,
com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to commit
or close the connection.
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
found when trying to get lock; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
2014-12-03 00:29:30,881 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:29:30,882 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
not found
2014-12-03 00:29:30,882 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
state Starting
2014-12-03 00:29:30,883 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
Starting
2014-12-03 00:29:30,883 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Unable to
complete AsyncJobVO {id:417, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:29:02 EST 2014}, job origin:414
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:29:30,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Complete async job-417,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:29:30,919 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-4:ctx-10fea75f work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:29:30,967 WARN [c.c.u.d.Merovingian2]
(HA-Worker-4:ctx-10fea75f work-2) Was unable to find lock for the
key vm_instance15 and thread id 139852089
2014-12-03 00:29:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-8:ctx-bc77e0da) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:29:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-61:ctx-de44c2a9) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:29:34,025 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-4:ctx-10fea75f work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:30:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-31:ctx-8417fc60) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:30:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-62:ctx-d118067c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:31:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-16:ctx-393a2cde) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:31:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-63:ctx-fd5d5e54) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:32:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-3:ctx-07ab5550) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:32:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-65:ctx-0345383e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:33:32,286 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-33:ctx-60af4723) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:33:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-66:ctx-0b95563b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:34:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-17:ctx-8e9c36a2) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:34:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-67:ctx-397176fb) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:35:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-9:ctx-3c967d9b) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:35:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-69:ctx-8bc1cfaf) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:36:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-35:ctx-8186e45f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:36:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-70:ctx-557403b1) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:37:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-18:ctx-8539dc77) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:37:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-71:ctx-61ea3221) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:38:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-4:ctx-0dc9a587) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:38:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-73:ctx-7fba328c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:39:31,006 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-0:ctx-7a356c50 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:39:31,051 WARN [c.c.u.d.Merovingian2]
(HA-Worker-0:ctx-7a356c50 work-5) Was unable to find lock for the
key vm_instance22 and thread id 84459079
2014-12-03 00:39:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-37:ctx-bc7a8f81) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:39:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-74:ctx-ebe4657b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:39:34,105 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-0:ctx-7a356c50 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:40:30,865 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:40:30,866 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 00:40:30,866 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Unable to
complete AsyncJobVO {id:426, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:40:02 EST 2014}, job origin:423
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:40:30,868 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Complete async job-426,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:40:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-19:ctx-338898ef) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:40:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-75:ctx-790ca221) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:41:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-10:ctx-0e2f52eb) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:41:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-77:ctx-f8287f50) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:42:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-39:ctx-1af09356) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:42:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-78:ctx-e1b2675f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:43:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-20:ctx-c9aea423) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:43:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-79:ctx-c459450e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:44:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-5:ctx-f7fee74e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:44:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-81:ctx-bc615860) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:45:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-41:ctx-4f287927) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:45:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-82:ctx-092eb488) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:46:27,281 WARN [o.a.c.alerts] (HA-1:ctx-b34246c5)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 00:46:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-21:ctx-62eca8f2) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:46:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-83:ctx-272e6406) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:47:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-11:ctx-23f28317) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:47:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-85:ctx-905ed14b) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:48:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-43:ctx-4a673855) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:48:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-86:ctx-1285bc0f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:49:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-22:ctx-60a35d36) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:49:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-87:ctx-687282ce) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
not found
2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
state Starting
2014-12-03 00:50:30,871 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
Starting
2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Unable to
complete AsyncJobVO {id:432, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 00:50:02 EST 2014}, job origin:429
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 00:50:30,873 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Complete async job-432,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 00:50:31,095 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-3:ctx-0dbe91b2 work-2) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 00:50:31,129 WARN [c.c.u.d.Merovingian2]
(HA-Worker-3:ctx-0dbe91b2 work-2) Was unable to find lock for the
key vm_instance15 and thread id 818653380
2014-12-03 00:50:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-6:ctx-2758188b) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:50:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-89:ctx-eb95a1c2) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:50:34,182 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-3:ctx-0dbe91b2 work-2) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 00:51:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-45:ctx-7be8bae6) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:51:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-90:ctx-3f76c1d7) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:52:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-23:ctx-e902ca80) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:52:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-91:ctx-4d56948f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:53:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-12:ctx-0876aaa4) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:53:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-93:ctx-c057ee22) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:54:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-47:ctx-8083fccf) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:54:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-94:ctx-56cebf3c) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:55:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-24:ctx-36c965de) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:55:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-95:ctx-86b1db16) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:56:27,241 WARN [o.a.c.alerts] (HA-2:ctx-627e96a3)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 00:56:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-1:ctx-6000a728) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:56:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-97:ctx-ffbe388e) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:57:32,279 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-49:ctx-a54e4fa0) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:57:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-98:ctx-34e48d1f) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:58:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-25:ctx-ffdd759e) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:58:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-99:ctx-0c1a9fa7) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 00:59:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-101:ctx-8442d937) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 00:59:32,963 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-51:ctx-9bd2b3cc) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:00:31,170 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(HA-Worker-2:ctx-f17ce3b5 work-5) Job is executed without a context, setup
psudo job for the executing thread
2014-12-03 01:00:31,208 WARN [c.c.u.d.Merovingian2]
(HA-Worker-2:ctx-f17ce3b5 work-5) Was unable to find lock for the
key vm_instance22 and thread id 52609881
2014-12-03 01:00:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-102:ctx-138c6123) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:00:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-26:ctx-3e13656a) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:00:34,263 WARN [c.c.h.HighAvailabilityManagerImpl]
(HA-Worker-2:ctx-f17ce3b5 work-5) Encountered unhandled exception during HA
process, reschedule retry
java.lang.RuntimeException: Job failed due to exception Work item not found,
We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
2014-12-03 01:01:32,280 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-103:ctx-334d11fc) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
transition the state but we're moving on because it's forced stop
2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
found
2014-12-03 01:01:32,856 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Invocation
exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
Starting
2014-12-03 01:01:32,857 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Rethrow
exception com.cloud.utils.exception.CloudRuntimeException: Work item not
found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 01:01:32,857 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Unable to
complete AsyncJobVO {id:438, userId: 1, accountId: 1, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
created: Wed Dec 03 01:01:03 EST 2014}, job origin:429
com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
2014-12-03 01:01:32,865 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Complete async job-438,
jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
2014-12-03 01:01:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-52:ctx-76332a49) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:02:32,282 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-105:ctx-b73de834) Unable to get current
status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:02:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-53:ctx-9b938dc5) Unable to get current status on
8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:03:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-27:ctx-28735fd7) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:03:32,967 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-107:ctx-e290c580) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:04:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-54:ctx-a53679d7) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:04:32,964 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-108:ctx-e1d3829f) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:05:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-55:ctx-65e26d72) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:05:32,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-110:ctx-b755e6ac) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
2014-12-03 01:06:27,239 WARN [o.a.c.alerts] (HA-3:ctx-b686e09d)
alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
message:: No usage server process running
2014-12-03 01:06:32,281 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-28:ctx-230f4b8f) Unable to get current status on
11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
2014-12-03 01:06:32,966 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgentCronJob-111:ctx-77cd282d) Unable to get current
status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
[root@CSTW1 ~]#
