Denis Finko created CLOUDSTACK-7757:
---------------------------------------
Summary: CloudStack don't remove VM from VMware
Key: CLOUDSTACK-7757
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7757
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Affects Versions: 4.2.1
Environment: VMware ESXi 5.1.0
Reporter: Denis Finko
Hello CloudStack community,
I have found a lot of old VMs that were removed from CloudStack but still
present in VMware. After that I created 6 similar VMs in CloudStack and tried
to remove then from user interface with 'Expunge' mark:
http://take.ms/vZ4AE
I was surprised that only 3 VMs from 6 were removed from VMware:
-------------------+-----------------------+------------------------------------------------+
name | removed time | Is VM was removed from VMware
-------------------+-----------------------+------------------------------------------------+
test-2010-3 | 07:46:53 | NO
-------------------+-----------------------+------------------------------------------------+
test-2010-4 | 07:49:45 | YES
-------------------+-----------------------+------------------------------------------------+
test-2010-5 | 07:55:32 | YES
-------------------+-----------------------+------------------------------------------------+
test-2110-1 | 07:57:08 | NO
-------------------+-----------------------+------------------------------------------------+
test-2110-2 | 08:00:44 | NO
-------------------+-----------------------+------------------------------------------------+
test-2110-3 | 08:03:06 | YES
-------------------+-----------------------+------------------------------------------------+
I have checked management-server.log for 'test-2110-2 ' and 'test-2110-3' VMs
and their looks similar:
logs for test-2110-2 (that VM wasn't removed from VMware environment):
2014-10-21 04:00:44,190 DEBUG [cloud.async.AsyncJobManagerImpl]
(ajp-20400-5:null) submit async job-7170 = [
b37ded66-33f7-4782-a36a-b7559d7d1462 ], details: AsyncJobVO {id:7170, userId:
2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId:
2185, cmd: org.apache.cloudstack.api.command.user.vm.DestroyVMCmd,
cmdOriginator: null, cmdInfo:
{"id":"b84498ba-bd20-4390-b239-0123fa566cac","response":"json","sessionkey":"vjRevp3+g1aH20gH0JkG8EVRSfg\u005d","cmdEventType":"VM.DESTROY","ctxUserId":"2","httpmethod":"GET","_":"1413878444174","ctxAccountId":"2","expunge":"true","ctxStartEventId":"407103"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 345048632606,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-10-21 04:00:44,190 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Executing
org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for job-7170 = [
b37ded66-33f7-4782-a36a-b7559d7d1462 ]
2014-10-21 04:00:44,213 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ])
Destroying vm VM[User|test-2110-2]
2014-10-21 04:00:44,213 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) VM is
already stopped: VM[User|test-2110-2]
2014-10-21 04:00:44,223 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) VM state
transitted from :Stopped to Destroyed with event: DestroyRequestedvm's original
host id: 22 new host id: null host id before state transition: null
2014-10-21 04:00:44,231 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Hosts's
actual total CPU: 67168 and CPU after applying overprovisioning: 67168
2014-10-21 04:00:44,231 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Hosts's
actual total RAM: 137391218688 and RAM after applying overprovisioning:
137391218688
2014-10-21 04:00:44,231 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) release
cpu from host: 22, old used: 56486,reserved: 1000, actual total: 67168, total
with overprovisioning: 67168; new used: 56486,reserved:0; movedfromreserved:
true,moveToReserveredfalse
2014-10-21 04:00:44,231 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) release
mem from host: 22, old used: 44023414784,reserved: 2147483648, total:
137391218688; new used: 44023414784,reserved:0; movedfromreserved:
true,moveToReserveredfalse
2014-10-21 04:00:44,256 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Stopped
called on VM[User|test-2110-2] but the state is Destroyed
2014-10-21 04:00:44,261 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) VM state
transitted from :Destroyed to Expunging with event: ExpungeOperationvm's
original host id: 22 new host id: null host id before state transition: null
2014-10-21 04:00:44,261 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ])
Destroying vm VM[User|test-2110-2]
2014-10-21 04:00:44,261 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Cleaning
up NICS
2014-10-21 04:00:44,264 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Cleaning
network for vm: 2185
2014-10-21 04:00:44,270 DEBUG [cloud.network.NetworkModelImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Service
SecurityGroup is not supported in the network id=207
2014-10-21 04:00:44,277 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Asking
VirtualRouter to release Nic[2615-2185-null-xx.xx.xx.xx]
2014-10-21 04:00:44,307 DEBUG [network.guru.DirectNetworkGuru]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ])
Deallocate network: networkId: 207, ip: xx.xx.xx.xx
2014-10-21 04:00:44,317 DEBUG [network.guru.DirectNetworkGuru]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) remove
nic 2615 secondary ip
2014-10-21 04:00:44,319 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Removed
nic id=2615
2014-10-21 04:00:44,320 DEBUG [cloud.network.NetworkManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Revoving
nic secondary ip entry ...
2014-10-21 04:00:44,323 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Cleaning
storage for vm: 2185
2014-10-21 04:00:44,390 DEBUG [agent.manager.ClusteredAgentAttache]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Seq
14-1013397135: Forwarding Seq 14-1013397135: { Cmd , MgmtId: 345048632606,
via: 14, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"d30ff69c-e473-470c-acbc-5aef797ba2ff","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"096b3b6e-481d-33b4-9b34-7178612a2535","id":2,"poolType":"VMFS","host":"VMFS
datastore:
/DC/SATA1","path":"/DC/SATA1","port":0}},"name":"ROOT-2185","size":10737418240,"path":"ROOT-2185","volumeId":2236,"vmName":"i-2-2185-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[SATA1]
i-2-2185-VM/ROOT-2185.vmdk\"]}","format":"OVA","id":2236,"hypervisorType":"VMware"}},"wait":0}}]
} to 345048630655
2014-10-21 04:00:44,564 DEBUG [agent.transport.Request]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Seq
14-1013397135: Received: { Ans: , MgmtId: 345048632606, via: 14, Ver: v1,
Flags: 10, { Answer } }
2014-10-21 04:00:44,571 INFO [storage.volume.VolumeServiceImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Volume
2236 is not referred anywhere, remove it from volumes table
2014-10-21 04:00:44,585 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Expunged
VM[User|test-2110-2]
2014-10-21 04:00:44,601 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Starting
cleaning up vm VM[User|test-2110-2] resources...
2014-10-21 04:00:44,608 DEBUG [network.firewall.FirewallManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) No
firewall rules are found for vm id=2185
2014-10-21 04:00:44,608 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Firewall
rules are removed successfully as a part of vm id=2185 expunge
2014-10-21 04:00:44,610 DEBUG [network.rules.RulesManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) No port
forwarding rules are found for vm id=2185
2014-10-21 04:00:44,611 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Port
forwarding rules are removed successfully as a part of vm id=2185 expunge
2014-10-21 04:00:44,612 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Removed
vm id=2185 from all load balancers as a part of expunge process
2014-10-21 04:00:44,613 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ])
Successfully cleaned up vm VM[User|test-2110-2] resources as a part of expunge
process
2014-10-21 04:00:44,625 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Complete
async job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ], jobStatus: 1,
resultCode: 0, result:
org.apache.cloudstack.api.response.UserVmResponse@58f21bb9
2014-10-21 04:00:44,627 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-28:job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]) Done
executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for job-7170 =
[ b37ded66-33f7-4782-a36a-b7559d7d1462 ]
2014-10-21 04:00:47,388 DEBUG [cloud.async.AsyncJobManagerImpl]
(ajp-20400-5:null) Async job-7170 = [ b37ded66-33f7-4782-a36a-b7559d7d1462 ]
completed
Please don't tell me that I am need to change expunge.delay or expunge.interval
in Global Settings. It's not helpful in this issue. I have already tried.
How I can remove old VMs from VMware environment via CloudStack?
Thank you!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
