shweta agarwal created CLOUDSTACK-4172:
------------------------------------------
Summary: Parallel deployment of vm fails when userdata is provided
Key: CLOUDSTACK-4172
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4172
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Environment: xenserver advance zone
Reporter: shweta agarwal
Priority: Blocker
Fix For: 4.2.0
Attachments: cloud-backup.dmp, management-server.log.tar.gz
Repro steps:
Set up - Advanced zone with 1 host (We all all the Vms to get deployed in this
host).
1. Create an account.
2. Create network for this account.
3. Deploy a Vm in this network , so that the router for this network is already
is running.
4. Deploy 30 - 50 Vms in parallel in this network by passing userdata.
called 30 vm deployment call with userdata and all vm deployment failed with
following exception
2013-08-07 21:53:34,410 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-87:job-836 =
[ 391486bb-e207-4e33-8eab-cc0993a9c49d ]) Applying userdata and password entry
in network Ntwk[207|Guest|8]
2013-08-07 21:53:34,422 DEBUG [agent.transport.Request]
(Job-Executor-87:job-836 = [ 391486bb-e207-4e33-8eab-cc0993a9c49d ]) Seq
1-122754475: Sending { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1, Flags:
100011,
[{"com.cloud.agent.api.routing.SavePasswordCommand":{"password":"fnirq_cnffjbeq","vmIpAddress":"10.1.1.136","vmName":"331d768a-33e6-4a00-b063-4c451e2acfee","executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.2.248","router.name":"r-219-VM"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.136","vmName":"331d768a-33e6-4a00-b063-4c451e2acfee","executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.2.248","router.name":"r-219-VM"},"wait":0}}]
}
2013-08-07 21:53:34,422 DEBUG [agent.transport.Request]
(Job-Executor-87:job-836 = [ 391486bb-e207-4e33-8eab-cc0993a9c49d ]) Seq
1-122754475: Executing: { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1, Flags:
100011,
[{"com.cloud.agent.api.routing.SavePasswordCommand":{"password":"fnirq_cnffjbeq","vmIpAddress":"10.1.1.136","vmName":"331d768a-33e6-4a00-b063-4c451e2acfee","executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.2.248","router.name":"r-219-VM"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.136","vmName":"331d768a-33e6-4a00-b063-4c451e2acfee","executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.2.248","router.name":"r-219-VM"},"wait":0}}]
}
2013-08-07 21:53:34,533 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-203:null) Seq 1-122754474: Executing request
2013-08-07 21:53:34,540 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-10:null) Ping from 3
2013-08-07 21:53:34,537 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-296:null) Seq 1-122754475: Executing request
2013-08-07 21:53:34,582 DEBUG [cloud.network.NetworkModelImpl]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ]) Service
SecurityGroup is not supported in the network id=207
2013-08-07 21:53:34,582 DEBUG [cloud.network.NetworkModelImpl]
(ApiServer-7:null) Service SecurityGroup is not supported in the network id=207
2013-08-07 21:53:34,586 DEBUG [cloud.network.NetworkModelImpl]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ]) Service
SecurityGroup is not supported in the network id=207
2013-08-07 21:53:34,951 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ]) Deploy
avoids pods: null, clusters: null, hosts: null
2013-08-07 21:53:34,956 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ])
DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_392cf6b8@1c529aa0
2013-08-07 21:53:34,956 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ]) Trying to
allocate a host and storage pools from dc:1, pod:null,cluster:null, requested
cpu: 128, requested ram: 134217728
2013-08-07 21:53:34,956 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ]) Is ROOT
volume READY (pool already allocated)?: No
2013-08-07 21:53:34,957 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ]) Searching
all possible resources under this Zone: 1
2013-08-07 21:53:34,957 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-118:job-839 = [ 8f1bcacd-91b6-4481-ba74-41664b43d283 ]) Listing
clusters in order of aggregate capacity, that have (atleast one host with)
enough CPU and RAM capacity under this Zone: 1
2013-08-07 21:53:35,403 DEBUG [cloud.vm.UserVmManagerImpl] (ApiServer-7:null)
Allocating in the DB for vm
2013-08-07 21:53:35,681 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(ApiServer-7:null) Allocating entries for VM:
VM[User|74cfaf6a-808e-4001-bbb7-a09a374b3733]
2013-08-07 21:53:35,682 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(ApiServer-7:null) Allocating nics for
VM[User|74cfaf6a-808e-4001-bbb7-a09a374b3733]
2013-08-07 21:53:35,683 DEBUG [cloud.network.NetworkManagerImpl]
(ApiServer-7:null) Allocating nic for vm
VM[User|74cfaf6a-808e-4001-bbb7-a09a374b3733] in network Ntwk[207|Guest|8] with
requested profile NicProfile[0-0-null-null-null
2013-08-07 21:53:36,050 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-203:null) Seq 1-122754474: Response Received:
2013-08-07 21:53:36,050 DEBUG [agent.transport.Request] (DirectAgent-203:null)
Seq 1-122754474: Processing: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"vm_data
failed","wait":0}}] }
207|Guest|8] with requested profile NicProfile[0-0-null-null-null
2013-08-07 21:53:36,050 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-203:null) Seq 1-122754474: Response Received:
2013-08-07 21:53:36,050 DEBUG [agent.transport.Request] (DirectAgent-203:null)
Seq 1-122754474: Processing: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"vm_data
failed","wait":0}}] }
2013-08-07 21:53:36,050 DEBUG [agent.transport.Request]
(Job-Executor-79:job-837 = [ b63e274a-ebfd-4a5c-9d27-6432fc1492f2 ]) Seq
1-122754474: Received: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1, Flags:
10, { Answer, Answer } }
2013-08-07 21:53:36,194 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-296:null) Seq 1-122754475: Response Received:
2013-08-07 21:53:36,264 DEBUG [agent.transport.Request] (DirectAgent-296:null)
Seq 1-122754475: Processing: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"vm_data
failed","wait":0}}] }
2013-08-07 21:53:36,264 DEBUG [agent.transport.Request]
(Job-Executor-87:job-836 = [ 391486bb-e207-4e33-8eab-cc0993a9c49d ]) Seq
1-122754475: Received: { Ans: , MgmtId: 7200344900649, via: 1, Ver: v1, Flags:
10, { Answer, Answer } }
2013-08-07 21:53:36,264 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-87:job-836 = [ 391486bb-e207-4e33-8eab-cc0993a9c49d ]) Unable to
contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: Unable to apply userdata and password entry on router
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3808)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyUserData(VirtualNetworkApplianceManagerImpl.java:2993)
at
com.cloud.network.element.VirtualRouterElement.addPasswordAndUserdata(VirtualRouterElement.java:941)
at
com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2035)
at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2141)
at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2082)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:932)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3408)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2968)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2954)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
