angeline shen created CLOUDSTACK-4125:
-----------------------------------------
Summary: [Upgrade] after upgrade 3.0.7 GA -> 4.2 deploy guest VM
fail
Key: CLOUDSTACK-4125
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4125
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Environment: MS 10.223.195.62 campo 4.2
host 10.223.51.3 XS 6.1
Reporter: angeline shen
Priority: Blocker
Fix For: 4.2.0
MS 10.223.195.62 build 3.0.7 GA upgrade 4.2 build host 10.223.51.3 XS 6.1
1. MS install 3.0.7 GA,
2. create advanced zone with SG
3. create several networks with different subnets and same vlan
vlan GW start IP end IP
----------------------------------------------------------------------------------------------------------
guest network 1250 10.223.125.1 10.223.125.13 10.223.125.18
1250-66to68 1250 10.223.125.65 10.223.125.66 10.223.125.68
1250-130to132 1250 10.223.125.129 10.223.125.130 10.223.125.132
4. start VMs on each of these networks
125.18 125.16 125.14 125.68 125.67 125.131 125.132
no ingress or egress rules :
VMs can ssh + ping other VMs in all subnets
VMs can egress internet
add ingress TCP icmp egress TCP egress
5. make sure VM work well
6. upgrade to 4.2
7. existing guest VM works welll
8. check multiple subnets
- add new IP range to subnet in network
- start new VM in network in second subnet - FAILed
2013-08-06 17:00:35,556 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Acquire lock
on VMTemplateStoragePool 2 with timeout 3600 seconds
2013-08-06 17:00:35,557 INFO [storage.volume.VolumeServiceImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) lock is
acquired for VMTemplateStoragePool 2
2013-08-06 17:00:35,559 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) failed to
create template on storage
com.cloud.utils.exception.CloudRuntimeException: Failed to update state
at
org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:164)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:433)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:538)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2488)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a
new state from null via CreateOnlyRequested
at com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:83)
at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:100)
at
org.apache.cloudstack.storage.datastore.ObjectInDataStoreManagerImpl.update(ObjectInDataStoreManagerImpl.java:258)
at
org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:162)
... 25 more
2013-08-06 17:00:35,579 INFO [storage.volume.VolumeServiceImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) releasing
lock for VMTemplateStoragePool 2
2013-08-06 17:00:35,579 ERROR [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Failed to
start instance VM[User|z1upgrade1250-130to132]
com.cloud.utils.exception.CloudRuntimeException: Failed to update state
at
org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:164)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:437)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:538)
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2488)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a
new state from null via OperationFailed
at com.cloud.utils.fsm.StateMachine2.getNextState(StateMachine2.java:83)
at com.cloud.utils.fsm.StateMachine2.transitTo(StateMachine2.java:100)
at
org.apache.cloudstack.storage.datastore.ObjectInDataStoreManagerImpl.update(ObjectInDataStoreManagerImpl.java:258)
at
org.apache.cloudstack.storage.image.store.TemplateObject.processEvent(TemplateObject.java:162)
2013-08-06 17:00:35,584 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Cleaning up
resources for the vm VM[User|z1upgrade1250-130to132] in Starting state
2013-08-06 17:00:35,585 DEBUG [agent.transport.Request] (Job-Executor-8:job-21
= [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Seq 1-1201344676: Sending { Cmd ,
MgmtId: 7264031213276, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-12-VM","wait":0}}]
}
2013-08-06 17:00:35,585 DEBUG [agent.transport.Request] (Job-Executor-8:job-21
= [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Seq 1-120
2013-08-06 17:00:35,852 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Destroying
vm VM[User|z1upgrade1250-130to132] as it failed to create on Host with Id:null
2013-08-06 17:00:35,866 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) VM state
transitted from :Stopped to Error with event: OperationFailedToErrorvm's
original host id: null new host id: null host id before state transition: null
2013-08-06 17:00:36,131 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null)
===START=== 10.216.133.70 -- GET
command=queryAsyncJobResult&jobId=f2595277-039b-4166-a8c3-01044c1b4cec&response=json&sessionkey=C2NtUAEudxOaSrj3%2B%2ByPxrOyXMo%3D&_=1375834266650
2013-08-06 17:00:36,158 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null)
===END=== 10.216.133.70 -- GET
command=queryAsyncJobResult&jobId=f2595277-039b-4166-a8c3-01044c1b4cec&response=json&sessionkey=C2NtUAEudxOaSrj3%2B%2ByPxrOyXMo%3D&_=1375834266650
2013-08-06 17:00:36,176 WARN [apache.cloudstack.alerts] (Job-Executor-8:job-21
= [ f2595277-039b-4166-a8c3-01044c1b4cec ]) alertType:: 8 // dataCenterId:: 1
// podId:: 1 // clusterId:: null // message:: Failed to deploy Vm with Id: 12,
on Host with Id: null
2013-08-06 17:00:36,209 INFO [user.vm.DeployVMCmd] (Job-Executor-8:job-21 = [
f2595277-039b-4166-a8c3-01044c1b4cec ])
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[User|z1upgrade1250-130to132]Scope=interface
com.cloud.dc.DataCenter; id=1
2013-08-06 17:00:36,209 INFO [user.vm.DeployVMCmd] (Job-Executor-8:job-21 = [
f2595277-039b-4166-a8c3-01044c1b4cec ]) Unable to create a deployment for
VM[User|z1upgrade1250-130to132]
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[User|z1upgrade1250-130to132]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-06 17:00:36,210 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-8:job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]) Complete
async job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ], jobStatus: 2,
resultCode: 530, result: Error Code: 533 Error text: Unable to create a
deployment for VM[User|z1upgrade1250-130to132]
2013-08-06 17:00:39,128 DEBUG [cloud.api.ApiServlet] (catalina-exec-23:null)
===START=== 10.216.133.70 -- GET
command=queryAsyncJobResult&jobId=f2595277-039b-4166-a8c3-01044c1b4cec&response=json&sessionkey=C2NtUAEudxOaSrj3%2B%2ByPxrOyXMo%3D&_=1375834269647
2013-08-06 17:00:39,141 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-23:null) Async job-21 = [ f2595277-039b-4166-a8c3-01044c1b4cec ]
completed
------------------------------------------------------------------------------------------
9. destroy 1 guest VM. Deploy guest VM : VM remain in STOPPED state
2013-08-06 17:24:57,230 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) No suitable
hosts found
2013-08-06 17:24:57,230 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) No suitable
hosts found under this Cluster: 1
2013-08-06 17:24:57,232 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Could not
find suitable Deployment Destinatio
n for this VM under any clusters, returning.
2013-08-06 17:24:57,232 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Searching
resources only under specified Pod: 1
2013-08-06 17:24:57,232 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Listing
clusters in order of aggregate capacity, that have (atleast one host with)
enough CPU and RAM capacity under this Pod: 1
2013-08-06 17:24:57,235 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Removing
from the clusterId list these clusters from avoid set: [1]
2013-08-06 17:24:57,235 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) No clusters
found after removing disabled clusters and clusters in avoid list, returning.
2013-08-06 17:24:57,280 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null)
===START=== 10.216.133.64 -- GET
command=queryAsyncJobResult&jobId=85b0ca6d-7586-4225-a9aa-b772ea99e3a8&response=json&sessionkey=kxfo3DkYEzBv44N8ynxFwuMn6bE%3D&_=1375835530701
2013-08-06 17:24:57,441 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's original
host id: null new host id: null host id before state transition: null
2013-08-06 17:24:57,447 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null)
===END=== 10.216.133.64 -- GET
command=queryAsyncJobResult&jobId=85b0ca6d-7586-4225-a9aa-b772ea99e3a8&response=json&sessionkey=kxfo3DkYEzBv44N8ynxFwuMn6bE%3D&_=1375835530701
2013-08-06 17:24:57,460 DEBUG [cloud.vm.UserVmManagerImpl]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Destroying
vm VM[User|187ff69f-e0d9-4364-bef6-86b59690b152] as it failed to create on Host
with Id:null
2013-08-06 17:24:58,427 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) VM state
transitted from :Stopped to Error with event: OperationFailedToErrorvm's
original host id: null new host id: null host id before state transition: null
2013-08-06 17:24:58,446 WARN [apache.cloudstack.alerts] (Job-Executor-1:job-24
= [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) alertType:: 8 // dataCenterId:: 1
// podId:: 1 // clusterId:: null // message:: Failed to deploy Vm with Id: 14,
on Host with Id: null
2013-08-06 17:24:58,476 INFO [user.vm.DeployVMCmd] (Job-Executor-1:job-24 = [
85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ])
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[User|187ff69f-e0d9-4364-bef6-86b59690b152]Scope=interface
com.cloud.dc.DataCenter; id=1
2013-08-06 17:24:58,476 INFO [user.vm.DeployVMCmd] (Job-Executor-1:job-24 = [
85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Unable to create a deployment for
VM[User|187ff69f-e0d9-4364-bef6-86b59690b152]
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[User|187ff69f-e0d9-4364-bef6-86b59690b152]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-06 17:24:58,477 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-1:job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ]) Complete
async job-24 = [ 85b0ca6d-7586-4225-a9aa-b772ea99e3a8 ], jobStatus: 2,
resultCode: 530, result: Error Code: 533 Error text: Unable to create a
deployment for VM[User|187ff69f-e0d9-4364-bef6-86b59690b152]
2013-08-06 17:24:59,563 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-9:null) SeqA 3-788: Processing Seq 3-788: { Cmd ,
MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-08-06 17:24:59,765 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-9:null) SeqA 3-788: Sending Seq 3-788: { Ans: , MgmtId:
7264031213276, via: 3, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
............
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
