[ https://issues.apache.org/jira/browse/CLOUDSTACK-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanjay Tripathi resolved CLOUDSTACK-6576. ----------------------------------------- Resolution: Won't Fix > No Error Handling while deploying GPU unsupported instances on a GPU enabled > host (Ex: Cent OS instance) > -------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-6576 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6576 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, XenServer > Affects Versions: 4.4.0 > Reporter: Sailaja Mada > Assignee: Sanjay Tripathi > Priority: Minor > Fix For: 4.4.0, 4.5.0 > > Attachments: logs1.zip > > > Steps: > 1. Install and configure Adv zone with GPU enabled XenServer 6.2.5 > 2. Try to deploy default Cent OS instance using GPU enabled offering( K2 - > 200 profile) > Observation: > 1. Xenserver does not support vGPU with Linux instances. > 2. Cloudstack throws deployment failure after throwing lot of exceptions by > XenServer. > Expected Results: > CloudStack should parse the failure provided by XS and provide proper > deployment failure as unsupported deployment. > 2014-05-05 16:38:50,730 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Seq 1-7036029992836286502: > Sending { Cmd , MgmtId: 55355881446856, via: 1(cldstk-R720-66), Ver: v1, > Flags: 100011, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":9,"name":"i-3-9-VM","bootloader":"PyGrub","type":"User","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":2147483648,"maxRam":2147483648,"arch":"x86_64","os":"CentOS > 5.6 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"1439f05b06e154d2","params":{},"uuid":"00069b3f-abf0-4f0d-a12c-6ae9d1a3ea81","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e92dcd30-3775-4762-bb83-b6c9d901644b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e8ccf63a-577b-3378-9741-16ce76631992","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.101","path":"/Sailaja/goletagpups1","port":2049,"url":"NetworkFilesystem://10.102.192.101/Sailaja/goletagpups1/?ROLE=Primary&STOREUUID=e8ccf63a-577b-3378-9741-16ce76631992"}},"name":"ROOT-9","size":21474836480,"path":"8d273d72-877c-4776-b4b5-9e52072ea45f","volumeId":11,"vmName":"i-3-9-VM","accountId":3,"format":"VHD","id":11,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"8d273d72-877c-4776-b4b5-9e52072ea45f","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"10.102.192.101","volumeSize":"21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"91973ed0-97f6-47bf-94ea-25751da81ae9","ip":"10.1.1.220","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:03:4b:00:07","dns1":"10.140.50.5","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://770","isolationUri":"vlan://770","isSecurityGroupEnabled":false}],"gpuDevice":{"gpuGroup":"Group > of NVIDIA Corporation GK104GL [GRID K2] GPUs","vgpuType":"GRID > K200"}},"hostIp":"10.102.192.66","executeInSequence":false,"wait":0}}] } > 2014-05-05 16:38:50,732 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Seq 1-7036029992836286502: > Executing: { Cmd , MgmtId: 55355881446856, via: 1(cldstk-R720-66), Ver: v1, > Flags: 100011, > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":9,"name":"i-3-9-VM","bootloader":"PyGrub","type":"User","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":2147483648,"maxRam":2147483648,"arch":"x86_64","os":"CentOS > 5.6 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"1439f05b06e154d2","params":{},"uuid":"00069b3f-abf0-4f0d-a12c-6ae9d1a3ea81","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e92dcd30-3775-4762-bb83-b6c9d901644b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e8ccf63a-577b-3378-9741-16ce76631992","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.101","path":"/Sailaja/goletagpups1","port":2049,"url":"NetworkFilesystem://10.102.192.101/Sailaja/goletagpups1/?ROLE=Primary&STOREUUID=e8ccf63a-577b-3378-9741-16ce76631992"}},"name":"ROOT-9","size":21474836480,"path":"8d273d72-877c-4776-b4b5-9e52072ea45f","volumeId":11,"vmName":"i-3-9-VM","accountId":3,"format":"VHD","id":11,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"8d273d72-877c-4776-b4b5-9e52072ea45f","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"10.102.192.101","volumeSize":"21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"91973ed0-97f6-47bf-94ea-25751da81ae9","ip":"10.1.1.220","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:03:4b:00:07","dns1":"10.140.50.5","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://770","isolationUri":"vlan://770","isSecurityGroupEnabled":false}],"gpuDevice":{"gpuGroup":"Group > of NVIDIA Corporation GK104GL [GRID K2] GPUs","vgpuType":"GRID > K200"}},"hostIp":"10.102.192.66","executeInSequence":false,"wait":0}}] } > 2014-05-05 16:38:50,732 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-296:ctx-251a7963) Seq 1-7036029992836286502: Executing request > 2014-05-05 16:38:50,743 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) 1. The VM i-3-9-VM is in Starting state. > 2014-05-05 16:38:50,780 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Created VM > 2b2d13d8-42ea-301d-6b65-0ad093195826 for i-3-9-VM > 2014-05-05 16:38:50,794 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Creating VGPU for of VGPU type: GRID K200 in > GPU group Group of NVIDIA Corporation GK104GL [GRID K2] GPUs for VM i-3-9-VM > 2014-05-05 16:38:50,794 DEBUG [c.c.h.x.r.XenServer620SP1Resource] > (DirectAgent-296:ctx-251a7963) Creating VGPU of VGPU type [ GRID K200 ] in > gpu groupGroup of NVIDIA Corporation GK104GL [GRID K2] GPUs for VM i-3-9-VM > 2014-05-05 16:38:50,818 DEBUG [c.c.h.x.r.XenServer620SP1Resource] > (DirectAgent-296:ctx-251a7963) Created VGPU of VGPU type [ GRID K200 ] for VM > i-3-9-VM > 2014-05-05 16:38:50,954 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) VBD 74e87e65-0066-f400-eb20-e322c0c64ebe > created for com.cloud.agent.api.to.DiskTO@61ba9994 > 2014-05-05 16:38:50,975 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) VBD d4e27c31-8ef4-49ca-661c-712f83568a03 > created for com.cloud.agent.api.to.DiskTO@21b8f5ac > 2014-05-05 16:38:50,975 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Creating VIF for i-3-9-VM on nic > [Nic:Guest-10.1.1.220-vlan://770] > 2014-05-05 16:38:51,000 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Found a network called > VLAN-0a144be7-8521-934e-ff53-c62c59e75d69-770 on host=10.102.192.66; > Network=fa62257b-e53e-1e82-69f0-963e63f853e0; > pif=4fb4b5fd-c11c-5827-8198-8861506bda56 > 2014-05-05 16:38:51,011 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Created a vif > c6b0df48-0a58-9640-5d23-302cf3c3d7b4 on 0 > 2014-05-05 16:38:52,033 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Task failed! Task record: > uuid: fe8bd262-4b47-9a13-ac8e-1c2cafbea4fb > nameLabel: Async.VM.start_on > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon May 05 15:28:17 IST 2014 > status: failure > residentOn: com.xensource.xenapi.Host@be7936ed > progress: 1.0 > type: <none/> > result: > errorInfo: [FEATURE_REQUIRES_HVM, vGPU- and GPU-passthrough needs > HVM] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > 2014-05-05 16:38:52,041 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Unable to start VM(i-3-9-VM) on > host(76520b97-5ce8-4c71-9572-b0834b62dc35) due to Task failed! Task record: > uuid: fe8bd262-4b47-9a13-ac8e-1c2cafbea4fb > nameLabel: Async.VM.start_on > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon May 05 15:28:17 IST 2014 > finished: Mon May 05 15:28:17 IST 2014 > status: failure > residentOn: com.xensource.xenapi.Host@be7936ed > progress: 1.0 > type: <none/> > result: > errorInfo: [FEATURE_REQUIRES_HVM, vGPU- and GPU-passthrough needs > HVM] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > Task failed! Task record: uuid: > fe8bd262-4b47-9a13-ac8e-1c2cafbea4fb > nameLabel: Async.VM.start_on > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon May 05 15:28:17 IST 2014 > finished: Mon May 05 15:28:17 IST 2014 > status: failure > residentOn: com.xensource.xenapi.Host@be7936ed > progress: 1.0 > type: <none/> > result: > errorInfo: [FEATURE_REQUIRES_HVM, vGPU- and GPU-passthrough needs > HVM] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3346) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3458) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1812) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:60) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:93) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:64) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-05 16:38:52,046 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) unable to destroy > task(com.xensource.xenapi.Task@d080c3e6) on > host(76520b97-5ce8-4c71-9572-b0834b62dc35) due to You gave an invalid object > reference. The object may have recently been deleted. The class parameter > gives the type of reference given, and the handle parameter echoes the bad > value given. > 2014-05-05 16:38:52,046 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Catch Exception: class > com.cloud.utils.exception.CloudRuntimeException due to > com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(i-3-9-VM) > on host(76520b97-5ce8-4c71-9572-b0834b62dc35) due to Task failed! Task > record: uuid: fe8bd262-4b47-9a13-ac8e-1c2cafbea4fb > nameLabel: Async.VM.start_on > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon May 05 15:28:17 IST 2014 > finished: Mon May 05 15:28:17 IST 2014 > status: failure > residentOn: com.xensource.xenapi.Host@be7936ed > progress: 1.0 > type: <none/> > result: > errorInfo: [FEATURE_REQUIRES_HVM, vGPU- and GPU-passthrough needs > HVM] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(i-3-9-VM) > on host(76520b97-5ce8-4c71-9572-b0834b62dc35) due to Task failed! Task > record: uuid: fe8bd262-4b47-9a13-ac8e-1c2cafbea4fb > nameLabel: Async.VM.start_on > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon May 05 15:28:17 IST 2014 > finished: Mon May 05 15:28:17 IST 2014 > status: failure > residentOn: com.xensource.xenapi.Host@be7936ed > progress: 1.0 > type: <none/> > result: > errorInfo: [FEATURE_REQUIRES_HVM, vGPU- and GPU-passthrough needs > HVM] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3477) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1812) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:60) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:93) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:64) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-05 16:38:52,048 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Unable to start i-3-9-VM due to > com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(i-3-9-VM) > on host(76520b97-5ce8-4c71-9572-b0834b62dc35) due to Task failed! Task > record: uuid: fe8bd262-4b47-9a13-ac8e-1c2cafbea4fb > nameLabel: Async.VM.start_on > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Mon May 05 15:28:17 IST 2014 > finished: Mon May 05 15:28:17 IST 2014 > status: failure > residentOn: com.xensource.xenapi.Host@be7936ed > progress: 1.0 > type: <none/> > result: > errorInfo: [FEATURE_REQUIRES_HVM, vGPU- and GPU-passthrough needs HVM] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3477) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1812) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:60) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:93) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:64) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-05 16:38:52,060 DEBUG [c.c.a.ApiServlet] > (catalina-exec-13:ctx-1d7ac3bf) ===START=== 10.144.6.6 -- GET > command=queryAsyncJobResult&jobId=48d1d975-c105-4823-99e1-6fc5f51f180e&response=json&sessionkey=TDXLIOmV8uKaEsb9Mr5o96exjw0%3D&_=1399288191562 > 2014-05-05 16:38:52,083 DEBUG [c.c.a.ApiServlet] > (catalina-exec-13:ctx-1d7ac3bf ctx-6f979a7b) ===END=== 10.144.6.6 -- GET > command=queryAsyncJobResult&jobId=48d1d975-c105-4823-99e1-6fc5f51f180e&response=json&sessionkey=TDXLIOmV8uKaEsb9Mr5o96exjw0%3D&_=1399288191562 > 2014-05-05 16:38:52,105 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Unable to clean up VBD due to > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:350) > at > com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456) > at com.xensource.xenapi.VBD.unplug(VBD.java:1059) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1542) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1892) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:60) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:93) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:64) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-05 16:38:52,112 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Unable to clean up VBD due to > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:350) > at > com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456) > at com.xensource.xenapi.VBD.unplug(VBD.java:1059) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1542) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1892) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:60) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:93) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:64) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-05 16:38:52,119 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) Unable to cleanup VIF > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:693) > at com.xensource.xenapi.Connection.dispatch(Connection.java:350) > at > com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456) > at com.xensource.xenapi.VIF.unplug(VIF.java:878) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1550) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1892) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:60) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:93) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:64) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-05 16:38:52,137 INFO [c.c.h.x.r.XenServer56Resource] > (DirectAgent-296:ctx-251a7963) Catch com.xensource.xenapi.Types$VifInUse: > failed to destory VLAN eth0 on host 76520b97-5ce8-4c71-9572-b0834b62dc35 due > to Network has active VIFs > 2014-05-05 16:38:52,137 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-296:ctx-251a7963) The VM is in stopped state, detected problem > during startup : i-3-9-VM > 2014-05-05 16:38:52,137 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-296:ctx-251a7963) Seq 1-7036029992836286502: Response Received: > 2014-05-05 16:38:52,139 DEBUG [c.c.a.t.Request] > (DirectAgent-296:ctx-251a7963) Seq 1-7036029992836286502: Processing: { Ans: > , MgmtId: 55355881446856, via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":9,"name":"i-3-9-VM","bootloader":"PyGrub","type":"User","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":2147483648,"maxRam":2147483648,"arch":"x86_64","os":"CentOS > 5.6 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"1439f05b06e154d2","params":{},"uuid":"00069b3f-abf0-4f0d-a12c-6ae9d1a3ea81","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e92dcd30-3775-4762-bb83-b6c9d901644b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e8ccf63a-577b-3378-9741-16ce76631992","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.101","path":"/Sailaja/goletagpups1","port":2049,"url":"NetworkFilesystem://10.102.192.101/Sailaja/goletagpups1/?ROLE=Primary&STOREUUID=e8ccf63a-577b-3378-9741-16ce76631992"}},"name":"ROOT-9","size":21474836480,"path":"8d273d72-877c-4776-b4b5-9e52072ea45f","volumeId":11,"vmName":"i-3-9-VM","accountId":3,"format":"VHD","id":11,"deviceId":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"8d273d72-877c-4776-b4b5-9e52072ea45f","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"10.102.192.101","volumeSize":"21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"91973ed0-97f6-47bf-94ea-25751da81ae9","ip":"10.1.1.220","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:03:4b:00:07","dns1":"10.140.50.5","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://770","isolationUri":"vlan://770","isSecurityGroupEnabled":false}],"gpuDevice":{"gpuGroup":"Group > of NVIDIA Corporation GK104GL [GRID K2] GPUs","vgpuType":"GRID > K200","groupDetails":{"Group of NVIDIA Corporation GK104GL [GRID K2] > GPUs":{"GRID K200":{"modelName":"GRID > K200","maxHeads":2,"videoRam":268435456,"maxResolutionX":1920,"maxResolutionY":1200,"maxVgpuPerGpu":8,"remainingCapacity":16,"maxCapacity":16},"passthrough":{"modelName":"passthrough","maxHeads":0,"videoRam":0,"maxResolutionX":0,"maxResolutionY":0,"maxVgpuPerGpu":1,"remainingCapacity":2,"maxCapacity":2},"GRID > K260Q":{"modelName":"GRID > K260Q","maxHeads":4,"videoRam":2013265920,"maxResolutionX":2560,"maxResolutionY":1600,"maxVgpuPerGpu":2,"remainingCapacity":4,"maxCapacity":4},"GRID > K240Q":{"modelName":"GRID > K240Q","maxHeads":2,"videoRam":1006632960,"maxResolutionX":2560,"maxResolutionY":1600,"maxVgpuPerGpu":4,"remainingCapacity":8,"maxCapacity":8}}}}},"_iqnToPath":{},"result":false,"details":"Unable > to start i-3-9-VM due to ","wait":0}}] } > 2014-05-05 16:38:52,139 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Seq 1-7036029992836286502: > Received: { Ans: , MgmtId: 55355881446856, via: 1, Ver: v1, Flags: 10, { > StartAnswer } } > 2014-05-05 16:38:52,142 INFO [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Unable to start VM on > Host[-1-Routing] due to Unable to start i-3-9-VM due to > 2014-05-05 16:38:52,145 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Cleaning up resources for > the vm VM[User|i-3-9-VM] in Starting state > 2014-05-05 16:38:52,148 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Seq 1-7036029992836286503: > Sending { Cmd , MgmtId: 55355881446856, via: 1(cldstk-R720-66), Ver: v1, > Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-3-9-VM","wait":0}}] > } > 2014-05-05 16:38:52,148 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Seq 1-7036029992836286503: > Executing: { Cmd , MgmtId: 55355881446856, via: 1(cldstk-R720-66), Ver: v1, > Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-3-9-VM","wait":0}}] > } > 2014-05-05 16:38:52,224 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Invocation exception, > caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to > create a deployment for VM[User|i-3-9-VM]Scope=interface > com.cloud.dc.DataCenter; id=1 > 2014-05-05 16:38:52,225 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-18:job-58/job-59 ctx-4a14e747) Rethrow exception > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-3-9-VM]Scope=interface com.cloud.dc.DataCenter; id=1 > 2014-05-05 16:38:52,226 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-18:job-58/job-59) Done with run of VM work job: > com.cloud.vm.VmWorkStart for VM 9, job origin: 58 > 2014-05-05 16:38:52,226 ERROR [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-18:job-58/job-59) Unable to complete AsyncJobVO {id:59, > userId: 3, accountId: 3, instanceType: null, instanceId: null, cmd: > com.cloud.vm.VmWorkStart, cmdInfo: > rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAADAAAAAAAAAAMAAAAAAAAACXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: > null, initMsid: 55355881446856, completeMsid: null, lastUpdated: null, > lastPolled: null, created: Mon May 05 16:38:46 IST 2014}, job origin:58 > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[User|i-3-9-VM]Scope=interface com.cloud.dc.DataCenter; id=1 > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:943) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5167) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5312) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-05 16:38:52,228 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-18:job-58/job-59) Complete async job-59, jobStatus: > FAILED, resultCode: 0, result: > rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAT0pvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gY3JlYXRlIGEgZGVwbG95bWVudCBmb3IgVk1bVXNlcnxpLTMtOS1WTV11cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAAA1zcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABEwACGZpbGVOYW1lcQB-AARMAAptZXRob2ROYW1lcQB-AAR4cAAAAHJ0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ACwAAAe90AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQB-AAsAAAAxdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AAsAAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfgALAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AGnQAD2NhbGxXaXRoQ29udGV4dHNxAH4ACwAAADVxAH4AHXEAfgAadAAOcnVuV2l0aENvbnRleHRzcQB-AAsAAAAudAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ABZxAH4AF3NxAH4ACwAAAcRxAH4AEXEAfgAScQB-ABdzcQB-AAsAAAHXdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ABtzcQB-AAsAAAEGdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAXc3EAfgALAAAEeXQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgALAAACZ3QALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ALHEAfgAXc3EAfgALAAAC6HQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAXc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAZ4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4AOHg > CCP should -- This message was sent by Atlassian JIRA (v6.3.4#6332)