I brought up a new setup with 4 hosts + management server following all the steps from my previous experimenting (single host + management server) but it is not working. I am not able to start an instance because the virtual router won't go. Any ideas?
Here are some logs: 2013-03-26 05:29:57,777 DEBUG [agent.transport.Request] (Job-Executor-4:job-12) Seq 8-113705002: Sending { Cmd , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 100111, [{"StartComm and":{"vm":{"id":4,"name":"r-4-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"speed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 6(64-b it)","bootArgs":" template=domP name=r-4-VM eth0ip=192.168.1.236 eth0mask=255.255.255.0 gateway=192.168.1.5 domain=cs1cloud.internal dhcprange=192.168.1.1 eth1ip=169.254.1.187 eth1mas k=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=202.56.230.6 dns2=202.56.230.5","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"vncPassword":"cd07ba516ab53c31","par ams":{},"uuid":"5057d314-b244-421f-8aa1-83a6de4faaf2","disks":[{"id":5,"name":"ROOT-4","mountPoint":"ext","path":"c12c4d67-124e-4542-b1c0-4243a6b94aba","size":2147483648,"type":"ROOT" ,"storagePoolType":"EXT","storagePoolUuid":"af137234-1dfb-6916-491f-9b72bb252a30","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"c0419883-0cc4-4f 21-8af2-980ac8813325","ip":"192.168.1.236","netmask":"255.255.255.0","gateway":"192.168.1.5","mac":"06:f0:74:00:00:48","dns1":"202.56.230.6","dns2":"202.56.230.5","broadcastType":"Nat ive","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"2fb0 9b08-a900-4fa7-aff3-898f15a82ddf","ip":"169.254.1.187","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:bb","broadcastType":"LinkLocal","type":"Control","isSecuri tyGroupEnabled":false}]},"wait":0}},{"check.CheckSshCommand":{"ip":"169.254.1.187","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"GetDomRVersionCmd":{"accessDetai ls":{"router.ip":"169.254.1.187","router.name":"r-4-VM"},"wait":0}},{}] } 2013-03-26 05:29:57,779 DEBUG [agent.transport.Request] (Job-Executor-4:job-12) Seq 8-113705002: Executing: { Cmd , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 100111, [{"StartC ommand":{"vm":{"id":4,"name":"r-4-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"speed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 6(6 4-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=192.168.1.236 eth0mask=255.255.255.0 gateway=192.168.1.5 domain=cs1cloud.internal dhcprange=192.168.1.1 eth1ip=169.254.1.187 eth1 mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=202.56.230.6 dns2=202.56.230.5","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"vncPassword":"cd07ba516ab53c31"," params":{},"uuid":"5057d314-b244-421f-8aa1-83a6de4faaf2","disks":[{"id":5,"name":"ROOT-4","mountPoint":"ext","path":"c12c4d67-124e-4542-b1c0-4243a6b94aba","size":2147483648,"type":"RO OT","storagePoolType":"EXT","storagePoolUuid":"af137234-1dfb-6916-491f-9b72bb252a30","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"c0419883-0cc4 -4f21-8af2-980ac8813325","ip":"192.168.1.236","netmask":"255.255.255.0","gateway":"192.168.1.5","mac":"06:f0:74:00:00:48","dns1":"202.56.230.6","dns2":"202.56.230.5","broadcastType":" Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"2 fb09b08-a900-4fa7-aff3-898f15a82ddf","ip":"169.254.1.187","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:bb","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"wait":0}},{"check.CheckSshCommand":{"ip":"169.254.1.187","port":3922,"interval":6,"retries":100,"name":"r-4-VM","wait":0}},{"GetDomRVersionCmd":{"accessDetails":{"router.ip":"169.254.1.187"," router.name":"r-4-VM"},"wait":0}},{}] } 2013-03-26 05:29:57,779 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq 8-113705002: Executing request 2013-03-26 05:29:58,041 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) 1. The VM r-4-VM is in Starting state. 2013-03-26 05:29:58,128 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-362:null) Seq 8-113705001: Response Received: 2013-03-26 05:29:58,128 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 8-113705001: Received: { Ans: , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } 2013-03-26 05:29:58,324 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Created VM 55d49fc7-e8b5-c39d-469f-94a8b4e3347f for r-4-VM 2013-03-26 05:29:58,445 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) PV args are -- quiet console=hvc0%template=domP%name=r-4-VM%eth0ip=192.168.1.236%eth0mask=255.255.255.0%gateway=192.168.1.5%domain=cs1cloud.internal%dhcprange=192.168.1.1%eth1ip=169.254.1.187%eth1mask=255.255.0.0%type=dhcpsrvr%disable_rp_filter=true%dns1=202.56.230.6%dns2=202.56.230.5 2013-03-26 05:29:58,495 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) VBD 693c0acd-b8f2-cfda-1083-a4115dd8a374 created for Vol[5|ROOT|c12c4d67-124e-4542-b1c0-4243a6b94aba|2147483648] 2013-03-26 05:29:58,817 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Creating VIF for r-4-VM on nic [Nic:Guest-192.168.1.236-vlan://untagged] 2013-03-26 05:29:58,864 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Created a vif ec727d55-9497-5d84-2274-abd6d98635bc on 0 2013-03-26 05:29:58,864 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Creating VIF for r-4-VM on nic [Nic:Control-169.254.1.187-null] 2013-03-26 05:29:58,913 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) already have a vif on dom0 for link local network 2013-03-26 05:29:58,969 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-45:null) Ping from 8 2013-03-26 05:29:59,279 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Created a vif 8fa1e517-1e83-15a0-7687-67f176eae01a on 1 2013-03-26 05:30:00,216 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-158:null) Seq 8-113704962: Executing request 2013-03-26 05:30:00,321 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Task failed! Task record: uuid: 3c38d58c-ba16-dd58-5f06-caf91f65c3d2 nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Tue Mar 26 05:29:59 IST 2013 finished: Tue Mar 26 05:29:59 IST 2013 status: FAILURE residentOn: com.xensource.xenapi.Host@4dfe54a9 progress: 1.0 type: <none/> result: errorInfo: [VM_REQUIRES_SR, OpaqueRef:3b5b2ac5-a6bb-b772-109c-a52d94cee5fe, OpaqueRef:785dc35b-f474-0998-21d5-22a090f9227e] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] 2013-03-26 05:30:00,329 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Unable to start VM(r-4-VM) on host(e4c6818a-6767-9a5a-a8b5-722095e72312) due to Task failed! Task record: uuid: 3c38d58c-ba16-dd58-5f06-caf91f65c3d2 nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Tue Mar 26 05:29:59 IST 2013 finished: Tue Mar 26 05:29:59 IST 2013 status: FAILURE residentOn: com.xensource.xenapi.Host@4dfe54a9 progress: 1.0 type: <none/> result: errorInfo: [VM_REQUIRES_SR, OpaqueRef:3b5b2ac5-a6bb-b772-109c-a52d94cee5fe, OpaqueRef:785dc35b-f474-0998-21d5-22a090f9227e] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] Task failed! Task record: uuid: 3c38d58c-ba16-dd58-5f06-caf91f65c3d2 nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Tue Mar 26 05:29:59 IST 2013 finished: Tue Mar 26 05:29:59 IST 2013 status: FAILURE residentOn: com.xensource.xenapi.Host@4dfe54a9 progress: 1.0 type: <none/> result: errorInfo: [VM_REQUIRES_SR, OpaqueRef:3b5b2ac5-a6bb-b772-109c-a52d94cee5fe, OpaqueRef:785dc35b-f474-0998-21d5-22a090f9227e] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3254) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1292) at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497) at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-03-26 05:30:00,338 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Catch Exception: class com.cloud.utils.exception.CloudRuntimeException due to com.cloud.utils.ex ception.CloudRuntimeException: Unable to start VM(r-4-VM) on host(e4c6818a-6767-9a5a-a8b5-722095e72312) due to Task failed! Task record: uuid: 3c38d58c-ba16-dd58-5f06-caf91f65c3d2 nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Tue Mar 26 05:29:59 IST 2013 finished: Tue Mar 26 05:29:59 IST 2013 status: FAILURE residentOn: com.xensource.xenapi.Host@4dfe54a9 progress: 1.0 type: <none/> result: errorInfo: [VM_REQUIRES_SR, OpaqueRef:3b5b2ac5-a6bb-b772-109c-a52d94cee5fe, OpaqueRef:785dc35b-f474-0998-21d5-22a090f9227e] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(r-4-VM) on host(e4c6818a-6767-9a5a-a8b5-722095e72312) due to Task failed! Task record: uuid: 3c38d58c-ba16-dd58-5f06-caf91f65c3d2 nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Tue Mar 26 05:29:59 IST 2013 finished: Tue Mar 26 05:29:59 IST 2013 status: FAILURE residentOn: com.xensource.xenapi.Host@4dfe54a9 progress: 1.0 type: <none/> result: errorInfo: [VM_REQUIRES_SR, OpaqueRef:3b5b2ac5-a6bb-b772-109c-a52d94cee5fe, OpaqueRef:785dc35b-f474-0998-21d5-22a090f9227e] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3265) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1292) at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497) at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-03-26 05:30:00,339 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Unable to start r-4-VM due to com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(r-4-VM) on host(e4c6818a-6767-9a5a-a8b5-722095e72312) due to Task failed! Task record: uuid: 3c38d58c-ba16-dd58-5f06-caf91f65c3d2 nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Tue Mar 26 05:29:59 IST 2013 finished: Tue Mar 26 05:29:59 IST 2013 status: FAILURE residentOn: com.xensource.xenapi.Host@4dfe54a9 progress: 1.0 type: <none/> result: errorInfo: [VM_REQUIRES_SR, OpaqueRef:3b5b2ac5-a6bb-b772-109c-a52d94cee5fe, OpaqueRef:785dc35b-f474-0998-21d5-22a090f9227e] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3265) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1292) at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497) at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-03-26 05:30:00,432 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Unable to clean up VBD due to You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:207) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) at com.xensource.xenapi.VBD.unplug(VBD.java:1054) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1092) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1353) at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497) at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-03-26 05:30:00,462 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Unable to clean up VBD due to You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:207) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) at com.xensource.xenapi.VBD.unplug(VBD.java:1054) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1092) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1353) at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497) at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-03-26 05:30:00,472 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-158:null) Seq 8-113704962: Response Received: 2013-03-26 05:30:00,472 DEBUG [agent.transport.Request] (DirectAgent-158:null) Seq 8-113704962: Processing: { Ans: , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 10, [{"Answer":{"result":true,"wait":0}}] } 2013-03-26 05:30:00,484 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Unable to cleanup VIF You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:207) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) at com.xensource.xenapi.VIF.unplug(VIF.java:804) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1100) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1353) at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497) at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-03-26 05:30:00,502 WARN [xen.resource.CitrixResourceBase] (DirectAgent-163:null) Unable to cleanup VIF You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:207) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) at com.xensource.xenapi.VIF.unplug(VIF.java:804) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1100) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1353) at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497) at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136) at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) 2013-03-26 05:30:00,515 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-163:null) The VM is in stopped state, detected problem during startup : r-4-VM 2013-03-26 05:30:00,515 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq 8-113705002: Cancelling because one of the answers is false and it is stop on error. 2013-03-26 05:30:00,515 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq 8-113705002: Response Received: 2013-03-26 05:30:00,516 DEBUG [agent.transport.Request] (DirectAgent-163:null) Seq 8-113705002: Processing: { Ans: , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 110, [{"StartAnswer":{"vm":{"id":4,"name":"r-4-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"speed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 6(64-bit)","bootArgs":" template=domP name=r-4-VM eth0ip=192.168.1.236 eth0mask=255.255.255.0 gateway=192.168.1.5 domain=cs1cloud.internal dhcprange=192.168.1.1 eth1ip=169.254.1.187 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=202.56.230.6 dns2=202.56.230.5","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"vncPassword":"cd07ba516ab53c31","params":{},"uuid":"5057d314-b244-421f-8aa1-83a6de4faaf2","disks":[{"id":5,"name":"ROOT-4","mountPoint":"ext","path":"c12c4d67-124e-4542-b1c0-4243a6b94aba","size":2147483648,"type":"ROOT","storagePoolType":"EXT","storagePoolUuid":"af137234-1dfb-6916-491f-9b72bb252a30","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"c0419883-0cc4-4f21-8af2-980ac8813325","ip":"192.168.1.236","netmask":"255.255.255.0","gateway":"192.168.1.5","mac":"06:f0:74:00:00:48","dns1":"202.56.230.6","dns2":"202.56.230.5","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"2fb09b08-a900-4fa7-aff3-898f15a82ddf","ip":"169.254.1.187","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:bb","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"result":false,"details":"Unable to start r-4-VM due to ","wait":0}}] } 2013-03-26 05:30:00,517 DEBUG [agent.manager.AgentAttache] (DirectAgent-163:null) Seq 8-113705002: No more commands found 2013-03-26 05:30:00,517 DEBUG [agent.transport.Request] (Job-Executor-4:job-12) Seq 8-113705002: Received: { Ans: , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 110, { StartAnswer } } 2013-03-26 05:30:00,521 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-4:job-12) Unable to start VM on Host[-8-Routing] due to Unable to start r-4-VM due to 2013-03-26 05:30:00,525 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-4:job-12) Cleaning up resources for the vm VM[DomainRouter|r-4-VM] in Starting state 2013-03-26 05:30:00,528 DEBUG [agent.transport.Request] (Job-Executor-4:job-12) Seq 8-113705003: Sending { Cmd , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-4-VM","wait":0}}] } 2013-03-26 05:30:00,528 DEBUG [agent.transport.Request] (Job-Executor-4:job-12) Seq 8-113705003: Executing: { Cmd , MgmtId: 159090355471823, via: 8, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-4-VM","wait":0}}] }