venkata swamybabu budumuru created CLOUDSTACK-4402:
------------------------------------------------------
Summary: [deleteStoragePool] There is no way to delete Primary
storage if the last host with which it was associated is already removed.
Key: CLOUDSTACK-4402
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4402
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Environment: commit id # 3c40e8bb3f6278f78c24c6317d513bd5ad599944
Reporter: venkata swamybabu budumuru
Fix For: 4.2.0
Steps to reproduce :
1. Have at least 1 advanced zone with XenServer Cluster
2. Doesn't actually matter whether VMs are deployed or not.
In my case the exact environment is as mentioned below
- 1 advanced zone
- 1 KVM cluster with 1 host
- 1 XenServer Cluster with 1 host
- All systemVMs are running on KVM host
3. place the XenServer into maintenance mode
2013-08-19 19:05:31,905 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null)
===START=== 10.252.192.25 -- GET
command=deleteHost&id=f498aa17-ed6c-4ca1-92e8-ef32c70c9290&forced=false&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919346034
2013-08-19 19:05:31,919 DEBUG [cloud.resource.ResourceManagerImpl]
(catalina-exec-22:null) Dispatching resource state event DELETE_HOST to
NetworkUsageManagerImpl_EnhancerByCloudStack_aa8b2ad3
2013-08-19 19:05:31,919 DEBUG [cloud.resource.ResourceManagerImpl]
(catalina-exec-22:null) Dispatching resource state event DELETE_HOST to
KvmServerDiscoverer_EnhancerByCloudStack_1647b3bf
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl]
(catalina-exec-22:null) Dispatching resource state event DELETE_HOST to
OvmDiscoverer_EnhancerByCloudStack_829e4aea
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl]
(catalina-exec-22:null) Dispatching resource state event DELETE_HOST to
VmwareServerDiscoverer_EnhancerByCloudStack_6ca85485
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl]
(catalina-exec-22:null) Dispatching resource state event DELETE_HOST to
PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_4b3fda74
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl]
(catalina-exec-22:null) Dispatching resource state event DELETE_HOST to
XcpServerDiscoverer_EnhancerByCloudStack_1934d343
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl]
(catalina-exec-22:null) Deleting Host: 9
Guid:d1f2d15f-17a5-4495-8f33-e8f5a5418413
2013-08-19 19:05:31,924 DEBUG [dc.dao.DataCenterIpAddressDaoImpl]
(catalina-exec-22:null) Releasing ip address: 10.147.40.31 data center 1
2013-08-19 19:05:31,926 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Host 9 is disconnecting with event Remove
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) The next status of agent 9is Removed, current status is
Up
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Deregistering link for 9 with state Removed
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Remove Agent : 9
2013-08-19 19:05:31,928 DEBUG [agent.manager.DirectAgentAttache]
(AgentTaskPool-9:null) Processing disconnect 9
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer_EnhancerByCloudStack_1934d343
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.deploy.DeploymentPlanningManagerImpl_EnhancerByCloudStack_dff95db9
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.network.NetworkManagerImpl_EnhancerByCloudStack_f18a6f4c
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.hypervisor.vmware.manager.VmwareManagerImpl_EnhancerByCloudStack_88ff6e46
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.network.security.SecurityGroupListener
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.storage.listener.StoragePoolMonitor
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.vm.ClusteredVirtualMachineManagerImpl_EnhancerByCloudStack_1e4b2050
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.storage.LocalStoragePoolListener
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.network.SshKeysDistriMonitor
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.network.router.VirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_b7bc8cf1
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.network.SshKeysDistriMonitor
2013-08-19 19:05:31,930 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_32a05514
2013-08-19 19:05:31,931 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.storage.upload.UploadListener
2013-08-19 19:05:31,931 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.storage.download.DownloadListener
2013-08-19 19:05:31,931 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.agent.manager.AgentMonitor
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.capacity.StorageCapacityListener
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.capacity.ComputeCapacityListener
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2013-08-19 19:05:31,932 DEBUG [cloud.network.NetworkUsageManagerImpl]
(AgentTaskPool-9:null) Disconnected called on 9 with status Removed
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Sending Disconnect to listener:
com.cloud.consoleproxy.ConsoleProxyListener
2013-08-19 19:05:31,932 DEBUG [cloud.host.Status] (AgentTaskPool-9:null)
Transition:[Resource state = Maintenance, Agent event = Remove, Host id = 9,
name = Rack1Pod1Host31]
2013-08-19 19:05:31,950 DEBUG [cloud.resource.ResourceState]
(catalina-exec-22:null) Resource state update: [id = 9; name = Rack1Pod1Host31;
old state = Maintenance; event = DeleteHost; new state = Disabled]
2013-08-19 19:05:31,955 DEBUG [cloud.host.Status] (AgentTaskPool-9:null) Agent
status update: [id = 9; name = Rack1Pod1Host31; old status = Up; event =
Remove; new status = Removed; old update count = 2; new update count = 3]
2013-08-19 19:05:31,955 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(AgentTaskPool-9:null) Notifying other nodes of to disconnect
2013-08-19 19:05:31,961 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null)
===END=== 10.252.192.25 -- GET
command=deleteHost&id=f498aa17-ed6c-4ca1-92e8-ef32c70c9290&forced=false&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919346034
4. remove the host first
5. Go the primary storage that is associated with Xenserver and place that into
maintenance mode
2013-08-19 19:06:44,458 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null)
===END=== 10.252.192.25 -- GET
command=enableStorageMaintenance&id=a1a1a150-6709-3d81-a414-4cee93e62168&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919418572
2013-08-19 19:06:44,461 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-63:job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]) Executing
org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
for job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]
2013-08-19 19:06:44,481 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-63:job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]) Complete
async job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ], jobStatus: 1,
resultCode: 0, result:
org.apache.cloudstack.api.response.StoragePoolResponse@5de2af25
2013-08-19 19:06:44,485 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-63:job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]) Done
executing
org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
for job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]
2013-08-19 19:06:47,504 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null)
===START=== 10.252.192.25 -- GET
command=queryAsyncJobResult&jobId=0b3ebe91-0101-4ba2-bb36-2327b9e7566f&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919421729
2013-08-19 19:06:47,512 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-20:null) Async job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]
completed
6. Now try to remove the host with forced=false
commanddeleteStoragePool&forced=false&id=a1a1a150-6709-3d81-a414-4cee93e62168responsejsonsessionkeyzJzyl0NdZA0QlMd8kSwWbynDxBI=
Observations :-
(i) It failed to delete with an error saying "Failed to delete storage pool on
host"
(ii) tried to delete it with forced=true and that as well failed with the same
error.
2013-08-19 19:06:55,677 ERROR [cloud.api.ApiServer] (catalina-exec-22:null)
unhandled exception executing api command: deleteStoragePool
com.cloud.utils.exception.CloudRuntimeException: Failed to delete storage pool
on host
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.deleteDataStore(CloudStackPrimaryDataStoreLifeCycleImpl.java:478)
at
com.cloud.storage.StorageManagerImpl.deletePool(StorageManagerImpl.java:937)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.admin.storage.DeletePoolCmd.execute(DeletePoolCmd.java:78)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:514)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:372)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:305)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-08-19 19:06:55,679 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null)
===END=== 10.252.192.25 -- GET
command=deleteStoragePool&id=a1a1a150-6709-3d81-a414-4cee93e62168&forced=false&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919429782
Attaching all the required logs along with db dump to the bug.
Expected Result:
(i) Either we should not let admin remove the hypervisor host while it has
primary storages associated and mounted on it
OR
(ii) we should let the deleteStoragePool with forced=true to delete the pool
forcefully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
