[
https://issues.apache.org/jira/browse/CLOUDSTACK-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chip Childers reopened CLOUDSTACK-2568:
---------------------------------------
reopening due to cherry-pick conflicts. let's only resolve when the code is in
the correct branch. Thanks Prachi!
> ACS41 regression in storage subsystem (seen with local storage and 2 or more
> hosts)
> -----------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-2568
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2568
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.1.0
> Environment: RHES64 as in OEL64. Install from RPM built from latest
> GIT on OEL64.
> 2 or more KVM hypervisors with local storage in one cluster that has one
> primary NFS storage pool.
> Reporter: Ove Ewerlid
> Assignee: Prachi Damle
> Priority: Blocker
> Fix For: 4.1.0
>
> Attachments: var-log-cloudstack-management.tar.gz
>
>
> ACS402 works with no issues when tested in exactly the setup where ACS41
> fails.
> Identical configuration (the same setup program is used for testing both
> versions).
> In ACS410 startVM fails if and only if the advanceStart: log line picks a
> poolID that is not valid.
> E.g., the poolID reported in this logline appears random across a large
> number of tests.
> If a poolID that can not be reached by the host selected for deployment, the
> startVM fails.
> This is blocking upgrade from 4.0 to 4.1 since there is no reliable way to
> start VMs that have been deployed. If a deployed VM fails to start, giving
> the startVM command multiple times, will eventually make the VM start.
> The more hosts there are, the less likely it is a startVM will succeed. It is
> less likely that the poolID is correct.
> The below log portion conveys how the VM has a "correct" Deployment
> Destination reported and the advanceStart reports a poolID that is different
> and since the selected hypervisor can not reach the poolID the startVM fails.
> The bug never triggers if there is only one KVM with local storage since the
> poolId can not be wrong, there is just one (and the NFS pool is always valid).
> -------------------
> 2013-05-20 06:49:29,477 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Found a potential host id: 1 name:
> vm3-net0-s0-14.test.devops and associated storage pools for this VM
> 2013-05-20 06:49:29,478 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Returning Deployment Destination:
> Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
> : Dest[Zone(1)-Pod(1)-Cluster(1)-Host\
> (1)-Storage(Volume(10|ROOT-->Pool(200), Volume(11|DATADISK-->Pool(200))]
> 2013-05-20 06:49:29,495 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-34:job-34) VM state transitted from :Stopped to Starting with
> event: StartRequestedvm's original host id: null new host id: null host id
> before state trans\
> ition: null
> 2013-05-20 06:49:29,495 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-34:job-34) Successfully transitioned to start state for
> VM[User|testvm-a] reservation id = e644d55e-3627-4395-9f89-639e6fc2f261
> 2013-05-20 06:49:29,502 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-34:job-34) Trying to deploy VM, vm has dcId: 1 and podId: null
> 2013-05-20 06:49:29,502 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-34:job-34) advanceStart: DeploymentPlan is provided, using
> dcId:1, podId: 1, clusterId: 1, hostId: 1, poolId: 201
> 2013-05-20 06:49:29,502 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-34:job-34) Deploy avoids pods: null, clusters: null, hosts: null
> 2013-05-20 06:49:29,504 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) DeploymentPlanner allocation algorithm: random
> 2013-05-20 06:49:29,504 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Trying to allocate a host and storage pools from
> dc:1, pod:1,cluster:1, requested cpu: 4000, requested ram: 2147483648
> 2013-05-20 06:49:29,504 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Is ROOT volume READY (pool already allocated)?: Yes
> 2013-05-20 06:49:29,504 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) DeploymentPlan has host_id specified, making no
> checks on this host, looks like admin test: 1
> 2013-05-20 06:49:29,505 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Looking for suitable pools for this host under zone:
> 1, pod: 1, cluster: 1
> 2013-05-20 06:49:29,506 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Checking suitable pools for volume (Id, Type):
> (10,ROOT)
> 2013-05-20 06:49:29,506 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Volume has pool(201) already allocated, checking if
> pool can be reused, poolId: null
> 2013-05-20 06:49:29,506 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) finding pool by id '201'
> 2013-05-20 06:49:29,507 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Planner need not allocate a pool for this volume
> since its READY
> 2013-05-20 06:49:29,507 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Checking suitable pools for volume (Id, Type):
> (11,DATADISK)
> 2013-05-20 06:49:29,507 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Volume has pool(201) already allocated, checking if
> pool can be reused, poolId: null
> 2013-05-20 06:49:29,507 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) finding pool by id '201'
> 2013-05-20 06:49:29,508 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Planner need not allocate a pool for this volume
> since its READY
> 2013-05-20 06:49:29,508 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Trying to find a potenial host and associated
> storage pools from the suitable host/pool lists for this VM
> 2013-05-20 06:49:29,508 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Checking if host: 1 can access any suitable storage
> pool for volume: DATADISK
> 2013-05-20 06:49:29,508 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Host: 1 cannot access pool: 201
> 2013-05-20 06:49:29,508 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Could not find a potential host that has associated
> storage pools from the suitable host/pool lists for this VM
> 2013-05-20 06:49:29,508 DEBUG [cloud.deploy.FirstFitPlanner]
> (Job-Executor-34:job-34) Cannnot deploy to specified host, returning.
> 2013-05-20 06:49:29,524 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-34:job-34) VM state transitted from :Starting to Stopped with
> event: OperationFailedvm's original host id: null new host id: null host id
> before state tran\
> sition: null
> 2013-05-20 06:49:29,533 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-34:job-34) Unexpected exception while executing
> org.apache.cloudstack.api.command.user.vm.StartVMCmd
> com.cloud.exception.InsufficientServerCapacityException: Unable to create a
> deployment for VM[User|testvm-a]Scope=interface com.cloud.dc.DataCenter; id=1
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:728)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3865)
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2573)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
