Hi Sm log is on the xenserver because you can know which mount point is set.
Sm log is under /var/log/SM.log James On Tuesday, July 14, 2015, Sonali Jadhav <[email protected]> wrote: > Any clue on this? > > I can understand that it's a problem while creating new VR. > > Catch Exception: class com.xensource.xenapi.Types$UuidInvalid due to The > uuid you supplied was invalid. > The uuid you supplied was invalid. > > I am not understanding which uuid is exactly invalid, I need help to trace > issue. > > /Sonali > > -----Original Message----- > From: Sonali Jadhav [mailto:[email protected] <javascript:;>] > Sent: Monday, July 13, 2015 1:42 PM > To: [email protected] <javascript:;> > Subject: RE: Urgent: VMs not migrated after putting Xenserver host in > maintenance mode > > Hi, > > That helped. I migrated vms and also in ACS it was syced correctly. Now > all my xenservers in pool are 6.5 . > > But I am having new problem, I am trying to make new vm with isolated > network. But its giving me following error, It looks like its problem while > creating VR. Also I observed that one host has 3 SRs which are > disconnected. I don't know why. It was like that after I rebooted server > with updated XS 6.5. > > 015-07-13 08:36:47,975 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Creating > monitoring services on VM[DomainRouter|r-97-VM] start... > 2015-07-13 08:36:47,982 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) > Reapplying dhcp entries as a part of domR VM[DomainRouter|r-97-VM] start... > 2015-07-13 08:36:47,984 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) > Reapplying vm data (userData and metaData) entries as a part of domR > VM[DomainRouter|r-97-VM] start... > 2015-07-13 08:36:48,035 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Seq > 4-5299892336484951126: Sending { Cmd , MgmtId: 59778234354585, via: > 4(SeSolXS02), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":97,"name":"r-97-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian > GNU/Linux 7(64-bit)","platformEmulator":"Debian Wheezy 7.0 > (64-bit)","bootArgs":" template=domP name=r-97-VM eth2ip=100.65.36.119 > eth2mask=255.255.255.192 gateway=100.65.36.65 eth0ip=10.1.1.1 > eth0mask=255.255.255.0 domain=cs17cloud.internal cidrsize=24 > dhcprange=10.1.1.1 eth1ip=169.254.0.120 eth1mask=255.255.0.0 type=router > disable_rp_filter=true dns1=8.8.8.8 > dns2=8.8.4.4","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"0R3TO+O9g+kGxMdtFbt0rw==","params":{},"uuid":"80b6edf0-7301-4985-b2a6-fae64636c5e8","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2fb465e2-f51f-4b46-8ec2-153fd843c6cf","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"876d490c-a1d4-3bfe-88b7-1bdb2479541b","id":1,"poolType":"NetworkFilesystem","host":"172.16.5.194","path":"/tank/primstore","port":2049,"url":"NetworkFilesystem:// > 172.16.5.194/tank/primstore/?ROLE=Primary&STOREUUID=876d490c-a1d4-3bfe-88b7-1bdb2479541b > "}},"name":"ROOT-97","size":2684354560,"path":"b9b23a67-9bfe-485a-906c-dfe8282fe868","volumeId":133,"vmName":"r-97-VM","accountId":23,"format":"VHD","provisioningType":"THIN","id":133,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"b9b23a67-9bfe-485a-906c-dfe8282fe868","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"172.16.5.194","volumeSize":"2684354560"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"f699a9b6-cc02-4e7e-805b-0005d69eadac","uuid":"1b5905ad-12b0-4594-be02-26aa753a640d","ip":"100.65.36.119","netmask":"255.255.255.192","gateway":"100.65.36.65","mac":"06:14:88:00:01:14","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://501","isolationUri":"vlan://501","isSecurityGroupEnabled":false,"name":"public"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"pxeDisable":true,"nicUuid":"52f8e291-c671-4bfe-b37b-9a0af82f09fd","uuid":"2a9f3c45-cdcf-4f39-a97c-ac29f1c21888","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:73:a2:00:02","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://714","isolationUri":"vlan://714","isSecurityGroupEnabled":false,"name":"guest"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"b1b8c3d6-e1e6-4575-8371-5c9b3e3a0c66","uuid":"527ed501-3b46-4d98-8e0a-d8d299870f32","ip":"169.254.0.120","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:78","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"172.16.5.198","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.120","port":3922,"interval":6,"retries":100,"name":"r-97-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{" > router.name > ":"r-97-VM","router.ip":"169.254.0.120"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"10.1.1.1"," > router.name > ":"r-97-VM","router.ip":"169.254.0.120"},"wait":0}},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":23,"publicIp":"100.65.36.119","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"broadcastUri":"vlan://501","vlanGateway":"100.65.36.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:af:70:00:01:14","networkRate":200,"trafficType":"Public","networkName":"public","newNic":false}],"accessDetails":{"zone.network.type":"Advanced"," > router.name > ":"r-97-VM","router.ip":"169.254.0.120","router.guest.ip":"10.1.1.1"},"wait":0}},{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":0,"srcIp":"","protocol":"all","revoked":false,"alreadyAdded":false,"sourceCidrList":[],"purpose":"Firewall","trafficType":"Egress","defaultEgressPolicy":false}],"accessDetails":{"router.guest.ip":"10.1.1.1","firewall.egress.default":"System","zone.network.type":"Advanced","router.ip":"169.254.0.120"," > router.name > ":"r-97-VM"},"wait":0}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":0,"service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":false},{"id":0,"service":"loadbalancing","processname":"haproxy","serviceName":"haproxy","servicePath":"/var/run/haproxy.pid","pidFile":"/var/run/haproxy.pid","isDefault":false},{"id":0,"service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":true},{"id":0,"service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2.pid","pidFile":"/var/run/apache2.pid","isDefault":true}],"accessDetails":{" > router.name > ":"r-97-VM","router.ip":"169.254.0.120","router.guest.ip":"10.1.1.1"},"wait":0}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"10.1.1.1"," > router.name":"r-97-VM","router.ip":"169.254.0.120"},"wait":0}}] } > 2015-07-13 08:36:48,036 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Seq > 4-5299892336484951126: Executing: { Cmd , MgmtId: 59778234354585, via: > 4(SeSolXS02), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":97,"name":"r-97-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian > GNU/Linux 7(64-bit)","platformEmulator":"Debian Wheezy 7.0 > (64-bit)","bootArgs":" template=domP name=r-97-VM eth2ip=100.65.36.119 > eth2mask=255.255.255.192 gateway=100.65.36.65 eth0ip=10.1.1.1 > eth0mask=255.255.255.0 domain=cs17cloud.internal cidrsize=24 > dhcprange=10.1.1.1 eth1ip=169.254.0.120 eth1mask=255.255.0.0 type=router > disable_rp_filter=true dns1=8.8.8.8 > dns2=8.8.4.4","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"0R3TO+O9g+kGxMdtFbt0rw==","params":{},"uuid":"80b6edf0-7301-4985-b2a6-fae64636c5e8","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2fb465e2-f51f-4b46-8ec2-153fd843c6cf","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"876d490c-a1d4-3bfe-88b7-1bdb2479541b","id":1,"poolType":"NetworkFilesystem","host":"172.16.5.194","path":"/tank/primstore","port":2049,"url":"NetworkFilesystem:// > 172.16.5.194/tank/primstore/?ROLE=Primary&STOREUUID=876d490c-a1d4-3bfe-88b7-1bdb2479541b > "}},"name":"ROOT-97","size":2684354560,"path":"b9b23a67-9bfe-485a-906c-dfe8282fe868","volumeId":133,"vmName":"r-97-VM","accountId":23,"format":"VHD","provisioningType":"THIN","id":133,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"b9b23a67-9bfe-485a-906c-dfe8282fe868","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"172.16.5.194","volumeSize":"2684354560"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"f699a9b6-cc02-4e7e-805b-0005d69eadac","uuid":"1b5905ad-12b0-4594-be02-26aa753a640d","ip":"100.65.36.119","netmask":"255.255.255.192","gateway":"100.65.36.65","mac":"06:14:88:00:01:14","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://501","isolationUri":"vlan://501","isSecurityGroupEnabled":false,"name":"public"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"pxeDisable":true,"nicUuid":"52f8e291-c671-4bfe-b37b-9a0af82f09fd","uuid":"2a9f3c45-cdcf-4f39-a97c-ac29f1c21888","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:73:a2:00:02","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://714","isolationUri":"vlan://714","isSecurityGroupEnabled":false,"name":"guest"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"b1b8c3d6-e1e6-4575-8371-5c9b3e3a0c66","uuid":"527ed501-3b46-4d98-8e0a-d8d299870f32","ip":"169.254.0.120","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:78","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"172.16.5.198","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.120","port":3922,"interval":6,"retries":100,"name":"r-97-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{" > router.name > ":"r-97-VM","router.ip":"169.254.0.120"},"wait":0}},{},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.guest.ip":"10.1.1.1"," > router.name > ":"r-97-VM","router.ip":"169.254.0.120"},"wait":0}},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":23,"publicIp":"100.65.36.119","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"broadcastUri":"vlan://501","vlanGateway":"100.65.36.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:af:70:00:01:14","networkRate":200,"trafficType":"Public","networkName":"public","newNic":false}],"accessDetails":{"zone.network.type":"Advanced"," > router.name > ":"r-97-VM","router.ip":"169.254.0.120","router.guest.ip":"10.1.1.1"},"wait":0}},{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":0,"srcIp":"","protocol":"all","revoked":false,"alreadyAdded":false,"sourceCidrList":[],"purpose":"Firewall","trafficType":"Egress","defaultEgressPolicy":false}],"accessDetails":{"router.guest.ip":"10.1.1.1","firewall.egress.default":"System","zone.network.type":"Advanced","router.ip":"169.254.0.120"," > router.name > ":"r-97-VM"},"wait":0}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":0,"service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":false},{"id":0,"service":"loadbalancing","processname":"haproxy","serviceName":"haproxy","servicePath":"/var/run/haproxy.pid","pidFile":"/var/run/haproxy.pid","isDefault":false},{"id":0,"service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":true},{"id":0,"service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2.pid","pidFile":"/var/run/apache2.pid","isDefault":true}],"accessDetails":{" > router.name > ":"r-97-VM","router.ip":"169.254.0.120","router.guest.ip":"10.1.1.1"},"wait":0}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.guest.ip":"10.1.1.1"," > router.name":"r-97-VM","router.ip":"169.254.0.120"},"wait":0}}] } > 2015-07-13 08:36:48,036 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-434:ctx-819aba7f) Seq 4-5299892336484951126: Executing request > 2015-07-13 08:36:48,043 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) 1. The VM r-97-VM is in Starting state. > 2015-07-13 08:36:48,065 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) Created VM > 14e931b3-c51d-fa86-e2d4-2e25059de732 for r-97-VM > 2015-07-13 08:36:48,069 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) PV args are -- quiet > console=hvc0%template=domP%name=r-97-VM%eth2ip=100.65.36.119%eth2mask=255.255.255.192%gateway=100.65.36.65%eth0ip=10.1.1.1%eth0mask=255.255.255.0%domain=cs17cloud.internal%cidrsize=24%dhcprange=10.1.1.1%eth1ip=169.254.0.120%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=8.8.8.8%dns2=8.8.4.4 > 2015-07-13 08:36:48,092 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) VBD e8612817-9d0c-2a6c-136f-5391831336e7 > created for com.cloud.agent.api.to.DiskTO@5b2138b > 2015-07-13 08:36:48,101 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) Catch Exception: class > com.xensource.xenapi.Types$UuidInvalid due to The uuid you supplied was > invalid. > The uuid you supplied was invalid. > at com.xensource.xenapi.Types.checkResponse(Types.java:1491) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:462) > at com.xensource.xenapi.VDI.getByUuid(VDI.java:341) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1580) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1784) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:489) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64) > at > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) > at > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2015-07-13 08:36:48,102 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) Unable to start r-97-VM due to > The uuid you supplied was invalid. > at com.xensource.xenapi.Types.checkResponse(Types.java:1491) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:462) > at com.xensource.xenapi.VDI.getByUuid(VDI.java:341) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1580) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1784) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:489) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64) > at > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) > at > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2015-07-13 08:36:48,124 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) Unable to clean up VBD due to > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:462) > at com.xensource.xenapi.VBD.unplug(VBD.java:1109) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1520) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1871) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:489) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64) > at > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) > at > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2015-07-13 08:36:48,128 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) Unable to clean up VBD due to > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:462) > at com.xensource.xenapi.VBD.unplug(VBD.java:1109) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1520) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1871) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:489) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64) > at > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) > at > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2015-07-13 08:36:48,129 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-434:ctx-819aba7f) The VM is in stopped state, detected problem > during startup : r-97-VM > 2015-07-13 08:36:48,129 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-434:ctx-819aba7f) Seq 4-5299892336484951126: Cancelling > because one of the answers is false and it is stop on error. > 2015-07-13 08:36:48,129 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-434:ctx-819aba7f) Seq 4-5299892336484951126: Response Received: > 2015-07-13 08:36:48,130 DEBUG [c.c.a.t.Request] > (DirectAgent-434:ctx-819aba7f) Seq 4-5299892336484951126: Processing: { > Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":97,"name":"r-97-VM","bootloader":"PyGrub","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian > GNU/Linux 7(64-bit)","platformEmulator":"Debian Wheezy 7.0 > (64-bit)","bootArgs":" template=domP name=r-97-VM eth2ip=100.65.36.119 > eth2mask=255.255.255.192 gateway=100.65.36.65 eth0ip=10.1.1.1 > eth0mask=255.255.255.0 domain=cs17cloud.internal cidrsize=24 > dhcprange=10.1.1.1 eth1ip=169.254.0.120 eth1mask=255.255.0.0 type=router > disable_rp_filter=true dns1=8.8.8.8 > dns2=8.8.4.4","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"0R3TO+O9g+kGxMdtFbt0rw==","params":{},"uuid":"80b6edf0-7301-4985-b2a6-fae64636c5e8","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2fb465e2-f51f-4b46-8ec2-153fd843c6cf","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"876d490c-a1d4-3bfe-88b7-1bdb2479541b","id":1,"poolType":"NetworkFilesystem","host":"172.16.5.194","path":"/tank/primstore","port":2049,"url":"NetworkFilesystem:// > 172.16.5.194/tank/primstore/?ROLE=Primary&STOREUUID=876d490c-a1d4-3bfe-88b7-1bdb2479541b"}},"name":"ROOT-97","size":2684354560,"path":"b9b23a67-9bfe-485a-906c-dfe8282fe868","volumeId":133,"vmName":"r-97-VM","accountId":23,"format":"VHD","provisioningType":"THIN","id":133,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"b9b23a67-9bfe-485a-906c-dfe8282fe868","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"172.16.5.194","volumeSize":"2684354560"}}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"f699a9b6-cc02-4e7e-805b-0005d69eadac","uuid":"1b5905ad-12b0-4594-be02-26aa753a640d","ip":"100.65.36.119","netmask":"255.255.255.192","gateway":"100.65.36.65","mac":"06:14:88:00:01:14","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://501","isolationUri":"vlan://501","isSecurityGroupEnabled":false,"name":"public"},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"pxeDisable":true,"nicUuid":"52f8e291-c671-4bfe-b37b-9a0af82f09fd","uuid":"2a9f3c45-cdcf-4f39-a97c-ac29f1c21888","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:73:a2:00:02","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://714","isolationUri":"vlan://714","isSecurityGroupEnabled":false,"name":"guest"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"b1b8c3d6-e1e6-4575-8371-5c9b3e3a0c66","uuid":"527ed501-3b46-4d98-8e0a-d8d299870f32","ip":"169.254.0.120","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:78","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"_iqnToPath":{},"result":false,"details":"Unable > to start r-97-VM due to ","wait":0}}] } > 2015-07-13 08:36:48,130 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Seq > 4-5299892336484951126: Received: { Ans: , MgmtId: 59778234354585, via: 4, > Ver: v1, Flags: 10, { StartAnswer } } > 2015-07-13 08:36:48,175 INFO [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Unable to > start VM on Host[-4-Routing] due to Unable to start r-97-VM due to > 2015-07-13 08:36:48,223 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Cleaning > up resources for the vm VM[DomainRouter|r-97-VM] in Starting state > 2015-07-13 08:36:48,230 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Seq > 4-5299892336484951127: Sending { Cmd , MgmtId: 59778234354585, via: > 4(SeSolXS02), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"r-97-VM","wait":0}}] > } > 2015-07-13 08:36:48,230 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-58f77d9c job-4353/job-4357 ctx-83fe75fb) Seq > 4-5299892336484951127: Executing: { Cmd , MgmtId: 59778234354585, via: > 4(SeSolXS02), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"r-97-VM","wait":0}}] > } > 2015-07-13 08:36:48,230 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-53:ctx-de9ca4c0) Seq 4-5299892336484951127: Executing request > > > /Sonali > > -----Original Message----- > From: Remi Bergsma [mailto:[email protected] <javascript:;>] > Sent: Saturday, July 11, 2015 5:34 PM > To: [email protected] <javascript:;> > Subject: Re: Urgent: VMs not migrated after putting Xenserver host in > maintenance mode > > Hi, > > Did you also set the 'removed' column back to NULL (instead of the > date/time it was originally deleted)? > > You can migrate directly from XenServer in 4.5.1, no problem. When the > hypervisor connects to CloudStack again it will report its running VMs and > update the data base. I guess there was a problem in 4.4.3 where > out-of-band migrations would cause a reboot of a router. Not sure if it is > also in 4.5.1. It's fixed in 4.4.4 and also in the upcoming 4.5.2. If your > remaining VMs are not routers, there is no issue. Otherwise you risk a > reboot (which is quite fast anyway). > > I'd first double check the disk offering, also check its tags etc. If that > works, then migrate in CloudStack (as it is supposed to work). If not, you > can do it directly from XenServer in order to empty your host and proceed > with the migration. Once the migration is done, fix any remaining issues. > > Hope this helps. > > Regards, > Remi > > > > On 11 jul. 2015, at 12:57, Sonali Jadhav <[email protected] > <javascript:;>> wrote: > > > > Hi I am using 4.5.1. That's why I am upgrading all xenservers to 6.5. > > > > I didn't knew that I can migrate vm from xenservers host itself. I > thought that would make cloudstack database inconsistent, since migration > is not initiated from cloudstack. > > > > And like I said before, those vms have compute offering which was > > deleted, but I "undeleted" them by setting status to "active" in > > disk_offering table > > > > Sent from my Sony Xperia(tm) smartphone > > > > > > ---- Remi Bergsma wrote ---- > > > > Hi Sonali, > > > > What version of CloudStack do you use? We can then look at the source at > line 292 of DeploymentPlanningManagerImpl.java If I look at master, it > indeed tries to do something with the compute offerings. Could you also > post its specs (print the result of the select query where you set the > field active). We might be able to tell what's wrong with it. > > > > As plan B, assuming you use a recent CloudStack version, you can use > > 'xe vm-migrate' to migrate VMs directly off of the hypervisor from the > > command line on the XenServer. Like this: xe vm-migrate vm=i-12-345-VM > > host=xen3 > > > > Recent versions of CloudStack will properly pick this up. When the VMS > are gone, the hypervisor will enter maintenance mode just fine. > > > > Regards, > > Remi > > > > > >> On 11 jul. 2015, at 09:42, Sonali Jadhav <[email protected] > <javascript:;>> wrote: > >> > >> Can anyone help me please? > >> > >> When I add xenserver host in maintenance, there are 3 VMs which are not > getting migrated to another host in cluster. > >> Other VMs were moved, but not these three. They both had computer > offering which was removed. But I undeleted those computer offerings, like > Andrija Panic suggested, changed their state to Active in > cloud.disk_offering table. > >> > >> But still I am seeing following errors, I am totally stuck, since I > have cluster of 4 xenservers, And I have upgraded 3 xenservers to 6.5, > except this one. And I can't reboot it for upgrade without moving these > instances to another host. > >> > >> [o.a.c.f.j.i.AsyncJobManagerImpl] (HA-Worker-2:ctx-68459b74 work-73) > >> Sync job-4090 execution on object VmWorkJobQueue.32 > >> 2015-07-09 14:27:00,908 INFO [c.c.h.HighAvailabilityManagerImpl] > >> (HA-Worker-3:ctx-6ee7e62f work-74) Processing > >> HAWork[74-Migration-34-Running-Scheduled] > >> 2015-07-09 14:27:01,147 WARN [o.a.c.f.j.AsyncJobExecutionContext] > >> (HA-Worker-3:ctx-6ee7e62f work-74) Job is executed without a context, > >> setup psudo job for the executing thread > >> 2015-07-09 14:27:01,162 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (HA-Worker-3:ctx-6ee7e62f work-74) Sync job-4091 execution on object > >> VmWorkJobQueue.34 > >> 2015-07-09 14:27:01,191 DEBUG [c.c.r.ResourceManagerImpl] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088 ctx-5921f0d2) Sent > >> resource event EVENT_PREPARE_MAINTENANCE_AFTER to listener > >> CapacityManagerImpl > >> 2015-07-09 14:27:01,206 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088 ctx-5921f0d2) Complete > >> async job-4088, jobStatus: SUCCEEDED, resultCode: 0, result: > >> org.apache.cloudstack.api.response.HostResponse/host/{"id":"c3c78959- > >> 6387-4cc9-8f59-23d44d2257a8","name":"SeSolXS03","state":"Up","disconn > >> ected":"2015-07-03T12:13:06+0200","type":"Routing","ipaddress":"172.1 > >> 6.5.188","zoneid":"1baf17c9-8325-4fa6-bffc-e502a33b578b","zonename":" > >> Solna","podid":"07de38ee-b63f-4285-819c-8abbdc392ab0","podname":"SeSo > >> lRack1","version":"4.5.1","hypervisor":"XenServer","cpusockets":2,"cp > >> unumber":24,"cpuspeed":2400,"cpuallocated":"0%","cpuused":"0%","cpuwi > >> thoverprovisioning":"57600.0","networkkbsread":0,"networkkbswrite":0, > >> "memorytotal":95574311424,"memoryallocated":0,"memoryused":13790400," > >> capabilities":"xen-3.0-x86_64 , xen-3.0-x86_32p , hvm-3.0-x86_32 , > >> hvm-3.0-x86_32p , > >> hvm-3.0-x86_64","lastpinged":"1970-01-17T06:39:19+0100","managementse > >> rverid":59778234354585,"clusterid":"fe15e305-5c11-4785-a13d-e4581e23f > >> 5e7","clustername":"SeSolCluster1","clustertype":"CloudManaged","islo > >> calstorageactive":false,"created":"2015-01-27T10:55:13+0100","events" > >> :"ManagementServerDown; AgentConnected; Ping; Remove; > >> AgentDisconnected; HostDown; ShutdownRequested; StartAgentRebalance; > >> PingTimeout","resourcestate":"PrepareForMaintenance","hypervisorversi > >> on":"6.2.0","hahost":false,"jobid":"7ad72023-a16f-4abf-84a3-83dd0e9f6 > >> bfd","jobstatus":0} > >> 2015-07-09 14:27:01,208 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088 ctx-5921f0d2) Publish > >> async job-4088 complete on message bus > >> 2015-07-09 14:27:01,208 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088 ctx-5921f0d2) Wake up > >> jobs related to job-4088 > >> 2015-07-09 14:27:01,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088 ctx-5921f0d2) Update db > >> status for job-4088 > >> 2015-07-09 14:27:01,211 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088 ctx-5921f0d2) Wake up > >> jobs joined with job-4088 and disjoin all subjobs created from job- > >> 4088 > >> 2015-07-09 14:27:01,386 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088) Done executing > >> org.apache.cloudstack.api.command.admin.host.PrepareForMaintenanceCmd > >> for job-4088 > >> 2015-07-09 14:27:01,389 INFO [o.a.c.f.j.i.AsyncJobMonitor] > >> (API-Job-Executor-107:ctx-4f5d495d job-4088) Remove job-4088 from job > >> monitoring > >> 2015-07-09 14:27:02,755 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (AsyncJobMgr-Heartbeat-1:ctx-1c99f7cd) Execute sync-queue item: > >> SyncQueueItemVO {id:2326, queueId: 251, contentType: AsyncJob, > >> contentId: 4091, lastProcessMsid: 59778234354585, lastprocessNumber: > >> 193, lastProcessTime: Thu Jul 09 14:27:02 CEST 2015, created: Thu Jul > >> 09 14:27:01 CEST 2015} > >> 2015-07-09 14:27:02,758 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (AsyncJobMgr-Heartbeat-1:ctx-1c99f7cd) Schedule queued job-4091 > >> 2015-07-09 14:27:02,810 INFO [o.a.c.f.j.i.AsyncJobMonitor] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Add job-4091 > >> into job monitoring > >> 2015-07-09 14:27:02,819 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Executing > >> AsyncJobVO {id:4091, userId: 1, accountId: 1, instanceType: null, > >> instanceId: null, cmd: com.cloud.vm.VmWorkMigrateAway, cmdInfo: > >> rO0ABXNyAB5jb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZUF3YXmt4MX4jtcEmwIAAUoAC > >> XNyY0hvc3RJZHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW > >> 50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cml > >> uZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAInQAGVZpcnR1YWxNYWNoaW5lTWFuYWdl > >> ckltcGwAAAAAAAAABQ, cmdVersion: 0, status: IN_PROGRESS, > >> processStatus: 0, resultCode: 0, result: null, initMsid: > >> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled: > >> null, created: Thu Jul 09 14:27:01 CEST 2015} > >> 2015-07-09 14:27:02,820 DEBUG [c.c.v.VmWorkJobDispatcher] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Run VM work > >> job: com.cloud.vm.VmWorkMigrateAway for VM 34, job origin: 3573 > >> 2015-07-09 14:27:02,822 DEBUG [c.c.v.VmWorkJobHandlerProxy] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091 ctx-744a984e) > >> Execute VM work job: > >> com.cloud.vm.VmWorkMigrateAway{"srcHostId":5,"userId":1,"accountId":1 > >> ,"vmId":34,"handlerName":"VirtualMachineManagerImpl"} > >> 2015-07-09 14:27:02,852 DEBUG [c.c.d.DeploymentPlanningManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091 ctx-744a984e) > >> Deploy avoids pods: [], clusters: [], hosts: [5] > >> 2015-07-09 14:27:02,855 ERROR [c.c.v.VmWorkJobHandlerProxy] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091 ctx-744a984e) > >> Invocation exception, caused by: java.lang.NullPointerException > >> 2015-07-09 14:27:02,855 INFO [c.c.v.VmWorkJobHandlerProxy] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091 ctx-744a984e) > >> Rethrow exception java.lang.NullPointerException > >> 2015-07-09 14:27:02,855 DEBUG [c.c.v.VmWorkJobDispatcher] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Done with run > >> of VM work job: com.cloud.vm.VmWorkMigrateAway for VM 34, job origin: > >> 3573 > >> 2015-07-09 14:27:02,855 ERROR [c.c.v.VmWorkJobDispatcher] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Unable to > complete AsyncJobVO {id:4091, userId: 1, accountId: 1, instanceType: null, > instanceId: null, cmd: com.cloud.vm.VmWorkMigrateAway, cmdInfo: > rO0ABXNyAB5jb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZUF3YXmt4MX4jtcEmwIAAUoACXNyY0hvc3RJZHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAInQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAABQ, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > result: null, initMsid: 59778234354585, completeMsid: null, lastUpdated: > null, lastPolled: null, created: Thu Jul 09 14:27:01 CEST 2015}, job > origin:3573 java.lang.NullPointerException > >> at > com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:292) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:2376) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:4517) > >> at sun.reflect.GeneratedMethodAccessor563.invoke(Unknown Source) > >> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:606) > >> at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > >> at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4636) > >> at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:537) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494) > >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) > >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:744) > >> 2015-07-09 14:27:02,863 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Complete async > >> job-4091, jobStatus: FAILED, resultCode: 0, result: > >> rO0ABXNyAB5qYXZhLmxhbmcuTnVsbFBvaW50ZXJFeGNlcHRpb25HpaGO_zHhuAIAAHhyA > >> BpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy > >> 5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMs > >> DAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0 > >> ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY > >> 2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0 > >> xpc3Q7eHBxAH4ACHB1cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz > >> 9IjkCAAB4cAAAABVzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUC > >> AARJAApsaW5lTnVtYmVyTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-A > >> AVMAAptZXRob2ROYW1lcQB-AAV4cAAAASR0AC5jb20uY2xvdWQuZGVwbG95LkRlcGxveW > >> 1lbnRQbGFubmluZ01hbmFnZXJJbXBsdAAiRGVwbG95bWVudFBsYW5uaW5nTWFuYWdlckl > >> tcGwuamF2YXQADnBsYW5EZXBsb3ltZW50c3EAfgALAAAJSHQAJmNvbS5jbG91ZC52bS5W > >> aXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wb > >> C5qYXZhdAAWb3JjaGVzdHJhdGVNaWdyYXRlQXdheXNxAH4ACwAAEaVxAH4AEXEAfgAScQ > >> B-ABNzcQB-AAv_____dAAmc3VuLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I > >> 1NjNwdAAGaW52b2tlc3EAfgALAAAAK3QAKHN1bi5yZWZsZWN0LkRlbGVnYXRpbmdNZXRo > >> b2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxA > >> H4AF3NxAH4ACwAAAl50ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2Quam > >> F2YXEAfgAXc3EAfgALAAAAa3QAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJ > >> veHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnNx > >> AH4ACwAAEhxxAH4AEXEAfgAScQB-ACFzcQB-AAsAAABndAAgY29tLmNsb3VkLnZtLlZtV > >> 29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2 > >> JzcQB-AAsAAAIZdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLml > >> tcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZh > >> dAAMcnVuSW5Db250ZXh0c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tY > >> W5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbn > >> RleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EAfgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWR > >> zdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQA > >> GkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNxAH4ACwAAAGd0AEBvcmcuY > >> XBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZW > >> RDb250ZXh0cQB-ADF0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-ADRxAH4AMXQ > >> ADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t > >> YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAtcQB-AC5zcQB-A > >> AsAAAHucQB-AChxAH4AKXEAfgAuc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW > >> 50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAyc3E > >> AfgALAAABBnQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVU > >> YXNrLmphdmFxAH4ALnNxAH4ACwAABHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlY > >> WRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcn > >> NxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3I > >> kV29ya2VycQB-AENxAH4ALnNxAH4ACwAAAuh0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhy > >> ZWFkLmphdmFxAH4ALnNyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlT > >> GlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAHeHIALGphdmEudXRpbC5Db2xsZWN0aW9ucy > >> RVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9 > >> Db2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNp > >> emV4cAAAAAB3BAAAAAB4cQB-AE94 > >> 2015-07-09 14:27:02,866 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Publish async > >> job-4091 complete on message bus > >> 2015-07-09 14:27:02,866 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Wake up jobs > >> related to job-4091 > >> 2015-07-09 14:27:02,866 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Update db > >> status for job-4091 > >> 2015-07-09 14:27:02,868 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Wake up jobs > >> joined with job-4091 and disjoin all subjobs created from job- 4091 > >> 2015-07-09 14:27:02,918 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Done executing > >> com.cloud.vm.VmWorkMigrateAway for job-4091 > >> 2015-07-09 14:27:02,926 INFO [o.a.c.f.j.i.AsyncJobMonitor] > >> (Work-Job-Executor-65:ctx-82ed9c8f job-3573/job-4091) Remove job-4091 > >> from job monitoring > >> 2015-07-09 14:27:02,979 WARN [c.c.h.HighAvailabilityManagerImpl] > >> (HA-Worker-3:ctx-6ee7e62f work-74) Encountered unhandled exception > during HA process, reschedule retry java.lang.NullPointerException > >> at > com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:292) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:2376) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:4517) > >> at sun.reflect.GeneratedMethodAccessor563.invoke(Unknown Source) > >> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:606) > >> at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > >> at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4636) > >> at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:537) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494) > >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) > >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:744) > >> 2015-07-09 14:27:02,980 INFO [c.c.h.HighAvailabilityManagerImpl] > >> (HA-Worker-3:ctx-6ee7e62f work-74) Rescheduling > >> HAWork[74-Migration-34-Running-Migrating] to try again at Thu Jul 09 > >> 14:37:16 CEST 2015 > >> 2015-07-09 14:27:03,008 DEBUG [c.c.a.m.AgentManagerImpl] > >> (AgentManager-Handler-14:null) SeqA 11-89048: Processing Seq > >> 11-89048: { Cmd , MgmtId: -1, via: 11, Ver: v1, Flags: 11, > >> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":8 > >> 0,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } > >> 2015-07-09 14:27:03,027 WARN [c.c.h.HighAvailabilityManagerImpl] > >> (HA-Worker-2:ctx-68459b74 work-73) Encountered unhandled exception > during HA process, reschedule retry java.lang.NullPointerException > >> at > com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:292) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:2376) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:4517) > >> at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) > >> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:606) > >> at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > >> at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4636) > >> at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:537) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494) > >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) > >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:744) > >> 2015-07-09 14:27:03,030 INFO [c.c.h.HighAvailabilityManagerImpl] > >> (HA-Worker-2:ctx-68459b74 work-73) Rescheduling > >> HAWork[73-Migration-32-Running-Migrating] to try again at Thu Jul 09 > >> 14:37:16 CEST 2015 > >> 2015-07-09 14:27:03,075 WARN [c.c.h.HighAvailabilityManagerImpl] > >> (HA-Worker-1:ctx-105d205a work-72) Encountered unhandled exception > during HA process, reschedule retry java.lang.NullPointerException > >> at > com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:292) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:2376) > >> at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:4517) > >> at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) > >> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:606) > >> at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > >> at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4636) > >> at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:537) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > >> at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494) > >> at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) > >> at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:744) > >> 2015-07-09 14:27:03,076 INFO [c.c.h.HighAvailabilityManagerImpl] > >> (HA-Worker-1:ctx-105d205a work-72) Rescheduling > >> HAWork[72-Migration-31-Running-Migrating] to try again at Thu Jul 09 > >> 14:37:16 CEST 2015 > >> 2015-07-09 14:27:03,165 DEBUG [c.c.a.m.AgentManagerImpl] > >> (AgentManager-Handler-14:null) SeqA 11-890 > >> > >> > >> > >> > >> > >> > >> > >> > >> /Sonali > >> > >> -----Original Message----- > >> From: Sonali Jadhav [mailto:[email protected] <javascript:;>] > >> Sent: Thursday, July 9, 2015 2:45 PM > >> To: [email protected] <javascript:;> > >> Subject: RE: VMs not migrated after putting Xenserver host in > >> maintenance mode > >> > >> Ignore this, I found problem. > >> > >> Though one question remains, from ACS If I try to migrate instance to > another host, it doesn't show upgraded host in list. Why is that ? > >> > >> /Sonali > >> > >> -----Original Message----- > >> From: Sonali Jadhav [mailto:[email protected] <javascript:;>] > >> Sent: Thursday, July 9, 2015 2:00 PM > >> To: [email protected] <javascript:;> > >> Subject: VMs not migrated after putting Xenserver host in maintenance > >> mode > >> > >> Hi, > >> > >> I am upgrading my xenserver from 6.2 to 6.5. I have cluster of 4 hosts. > I have managed to upgrade two of the hosts. I added 3d host in maintenance > mode from ACS, some VMs were moved to another host, but 4 VMs did not got > moved to another host. I saw few errors in logs. > >> > >> http://pastebin.com/L7TjLHwq > >> > >> http://pastebin.com/i1EGnEJr > >> > >> One more thing I observed is that, from ACS If I try to migrate vm to > another host, it doesn't show upgraded host in list. Why is that ? > >> > >> /Sonali > > > >
