Sailaja Mada created CLOUDSTACK-4131:
----------------------------------------
Summary: [VMWARE] ROOT volumes are not getting deleted when VM's
(User/Sytem) are failed to deploy/Start
Key: CLOUDSTACK-4131
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4131
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, VMware
Affects Versions: 4.2.0
Reporter: Sailaja Mada
Priority: Critical
Setup : VMWARE ESXi servers
Steps:
1. Configure Adv Zone with VMWARE Clutser
2. There is a issue with public traffic label name so System VM's failed start
while creating Public port groups. (Its one of the scenario where system VM's
can fail to start)
3. It started creating System VM's again and again and every time it failed
with same error
Observation:
1. When ever there is failure to create instances we are deleting the ROOT
volumes , We are narking them as Destory state but Removed is not updated .
2. These volumes are physically present in the primary storage and on the ESXi
server
3. After having the actual issue solved also we will not be able to get the
system VM's running as it fails saying ROOT Volume already exists :
2013-08-07 12:20:54,024 DEBUG [cloud.storage.VolumeManagerImpl]
(consoleproxy-1:null) Checking if we need to prepare 1 volumes for
VM[ConsoleProxy|v-2-VM]
2013-08-07 12:20:54,049 DEBUG [storage.image.TemplateDataFactoryImpl]
(consoleproxy-1:null) template 8 is already in store:1, type:Image
2013-08-07 12:20:54,059 DEBUG [storage.image.TemplateDataFactoryImpl]
(consoleproxy-1:null) template 8 is already in store:1, type:Primary
2013-08-07 12:20:54,097 DEBUG [storage.motion.AncientDataMotionStrategy]
(consoleproxy-1:null) copyAsync inspecting src type TEMPLATE copyAsync
inspecting dest type VOLUME
2013-08-07 12:20:54,119 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 2-1287389204: Sending { Cmd , MgmtId: 187767034175903, via: 2, Ver: v1,
Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"1d8a03ea04a03db296074bcf3fb39c35","origUrl":"http://download.cloud.com/templates/4.2/systemvmtemplate-4.2-vh7.ova","uuid":"4a64b93e-ff20-11e2-a5c2-aac5eb5f619f","id":8,"format":"OVA","accountId":1,"checksum":"8fde62b1089e5844a9cd3b9b953f9596","hvm":false,"displayText":"SystemVM
Template
(vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"a24ea7dc-cd9b-35bd-a7d4-9e8b4b8614c5","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/finalps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d8c4189f-fcaa-4942-84df-9c05f29fc408","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"a24ea7dc-cd9b-35bd-a7d4-9e8b4b8614c5","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/finalps1","port":2049}},"name":"ROOT-2","size":0,"volumeId":15,"vmName":"v-2-VM","accountId":1,"format":"OVA","id":15,"hypervisorType":"None"}},"executeInSequence":false,"wait":0}}]
}
2013-08-07 12:20:54,123 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 2-1287389204: Executing: { Cmd , MgmtId: 187767034175903, via: 2, Ver: v1,
Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"1d8a03ea04a03db296074bcf3fb39c35","origUrl":"http://download.cloud.com/templates/4.2/systemvmtemplate-4.2-vh7.ova","uuid":"4a64b93e-ff20-11e2-a5c2-aac5eb5f619f","id":8,"format":"OVA","accountId":1,"checksum":"8fde62b1089e5844a9cd3b9b953f9596","hvm":false,"displayText":"SystemVM
Template
(vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"a24ea7dc-cd9b-35bd-a7d4-9e8b4b8614c5","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/finalps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d8c4189f-fcaa-4942-84df-9c05f29fc408","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"a24ea7dc-cd9b-35bd-a7d4-9e8b4b8614c5","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/finalps1","port":2049}},"name":"ROOT-2","size":0,"volumeId":15,"vmName":"v-2-VM","accountId":1,"format":"OVA","id":15,"hypervisorType":"None"}},"executeInSequence":false,"wait":0}}]
}
2013-08-07 12:20:54,125 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-11:null) Seq 2-1287389204: Executing request
2013-08-07 12:20:54,297 INFO [storage.resource.VmwareStorageProcessor]
(DirectAgent-11:10.102.192.18) creating linked clone from template
2013-08-07 12:20:55,378 ERROR [storage.resource.VmwareStorageProcessor]
(DirectAgent-11:10.102.192.18) CreateCommand failed due to Exception:
java.lang.RuntimeException
Message: The name 'ROOT-2' already exists.
java.lang.RuntimeException: The name 'ROOT-2' already exists.
at
com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at
com.cloud.hypervisor.vmware.mo.VirtualMachineMO.createLinkedClone(VirtualMachineMO.java:641)
at
com.cloud.storage.resource.VmwareStorageProcessor.createVMLinkedClone(VmwareStorageProcessor.java:251)
at
com.cloud.storage.resource.VmwareStorageProcessor.cloneVolumeFromBaseTemplate(VmwareStorageProcessor.java:368)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:73)
at
com.cloud.storage.resource.VmwareStorageSubsystemCommandHandler.execute(VmwareStorageSubsystemCommandHandler.java:109)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:564)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-07 12:20:55,383 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-11:null) Seq 2-1287389204: Response Received:
2013-08-07 12:20:55,383 DEBUG [agent.transport.Request] (DirectAgent-11:null)
Seq 2-1287389204: Processing: { Ans: , MgmtId: 187767034175903, via: 2, Ver:
v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"java.lang.RuntimeException:
The name 'ROOT-2' already exists.","wait":0}}] }
2013-08-07 12:20:55,383 DEBUG [agent.transport.Request] (consoleproxy-1:null)
Seq 2-1287389204: Received: { Ans: , MgmtId: 187767034175903, via: 2, Ver: v1,
Flags: 10, { CopyCmdAnswer } }
2013-08-07 12:20:55,425 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(consoleproxy-1:null) Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@37655fe4), no need
to delete from object in store ref table
2013-08-07 12:20:55,426 DEBUG [cloud.storage.VolumeManagerImpl]
(consoleproxy-1:null) Unable to create
Vol[15|vm=2|ROOT]:java.lang.RuntimeException: The name 'ROOT-2' already exists.
2013-08-07 12:20:55,426 INFO [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Unable to contact resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
