[
https://issues.apache.org/jira/browse/CLOUDSTACK-7651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Likitha Shetty resolved CLOUDSTACK-7651.
----------------------------------------
Resolution: Fixed
Fixed by
[https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=9866c648eb5e6ce6ded1b6dcadbb77240fe0e683]
> Various operation intermittently fail in VMware with 'he object has already
> been deleted or has not been completely created' exceptions
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7651
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7651
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: VMware
> Affects Versions: 4.5.0
> Reporter: Likitha Shetty
> Assignee: Likitha Shetty
> Priority: Critical
> Fix For: 4.5.0
>
>
> +Steps to reproduce+
> 1. Setup VMware 4.3.0.1 with 1 CCP Management server.
> 2. Create 2 VMware DCs in 2 different vCenters.
> 3. Add 2 zones in CCP corresponding to the two DCs.
> 4. Do basic VM/Volume lifecycle operations.
> Keep monitoring the logs for the following error -
> *javax.xml.ws.soap.SOAPFaultException: The object has already been deleted or
> has not been completely created*
> For e.g.
> {noformat}
> javax.xml.ws.soap.SOAPFaultException
> Message: The object has already been deleted or has not been completely
> created
> javax.xml.ws.soap.SOAPFaultException: The object has already been deleted or
> has not been completely created
> at
> com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
> at
> com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:119)
> at
> com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:108)
> at
> com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
> at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:110)
> at com.sun.proxy.$Proxy389.retrieveProperties(Unknown Source)
> at
> com.cloud.hypervisor.vmware.util.VmwareClient.retrieveMoRefProperties(VmwareClient.java:308)
> at
> com.cloud.hypervisor.vmware.util.VmwareClient.getDynamicProperty(VmwareClient.java:263)
> at
> com.cloud.hypervisor.vmware.mo.HostMO.getHostDatastoreSystemMO(HostMO.java:166)
> at
> com.cloud.hypervisor.vmware.mo.HostMO.findDatastore(HostMO.java:889)
> at
> com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.findDatastoreWithBackwardsCompatibility(HypervisorHostHelper.java:131)
> at
> com.cloud.storage.resource.VmwareStorageProcessor.deleteVolume(VmwareStorageProcessor.java:1481)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:120)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:54)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:598)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:215)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:50)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:47)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:701)
> {noformat}
> +Root Cause Analysis+
> The thread local context which is used to handle connections to a vCenter is
> being re-used intermittently. And when it is re-used, if the existing context
> corresponds to the other vCenter then the operation being executed will fail.
> The re-use is happening because after the PingTask completion we are not
> clearing the ThreadLocal context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
