Re: Error Creating VMs and viewing hosts

Mon, 15 Oct 2012 10:17:39 -0700 

On 10/15/2012 12:00 PM, Alena Prokharchyk wrote:

Nic, looks like when storage pool was removed, cloudStack kept references
host-> removed storage pool in the DB - its a bug.

To workaround on your current setup:

* save the db dump of cloud db just in case
* delete all the stale storageHost refs to remove:

"delete from storage_pool_host_ref where pool_id not in (select id from
storage_pool)"

Now listHosts should execute w/o any problems. Let me know if it works.
It worked, but there are some fk constraints, so setting deleted=now() on offline primary storage in storage_pool isn't enough. I deleted the offline primary storage from storage_pool then deleted associated rows from storage_pool_host_ref, then restarted cloud-management, and hosts are now showing up; 


-Alena.


On 10/15/12 9:45 AM, "Nik Martin" <[email protected]> wrote:


On 10/15/2012 11:40 AM, Alena Prokharchyk wrote:

Nik, the error you see, doesn't state that the VM migration failed. NPE
happens on listHosts command, when we are building the response and
trying to figure out if local storage is enabled on the host.

What version of cloudStack you are running on? Did you remove any of
the primary storages in your setup?

-Alena.



Alena,

This is a 3.02 CS with Xenserver hosts.  I have removed and re-added
primary storage, to work around bug 251, which takes an entire cloud
offline if any primary storage is put into maintenance mode and
shutdown.  I guess the fallout from that is more serious than earlier
expected. I'll need further direction on how to recover from bug 251,
which knocks me entirely offline.

Nik




From: Nik Martin
<[email protected]<mailto:[email protected]>>
Reply-To: "'cloudstack-users@incubator. org'"
<[email protected]<mailto:cloudstack-users@incubator.
apache.org>>
To: "'cloudstack-users@incubator. org'"
<[email protected]<mailto:cloudstack-users@incubator.
apache.org>>
Subject: Error Creating VMs and viewing hosts

I migrated some VMs around to different storage tags (I have two primary
SANs, HD and SSD), and at some point, a migration failed, with this
error:


2012-10-15 09:34:32,399 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(StatsCollector-1:null) Cleanup succeeded. Details null
2012-10-15 09:34:32,408 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-188:null) Seq 4-857023789: Executing request
2012-10-15 09:34:32,626 ERROR [cloud.api.ApiDispatcher]
(catalina-exec-2:null) Exception while executing ListHostsCmd:
java.lang.NullPointerException
at

com.cloud.storage.StorageManagerImpl.isLocalStorageActiveOnHost(StorageMa
nagerImpl.java:409)
at
com.cloud.api.ApiDBUtils.isLocalStorageActiveOnHost(ApiDBUtils.java:340)
at

com.cloud.api.ApiResponseHelper.createHostResponse(ApiResponseHelper.java
:640)
at com.cloud.api.commands.ListHostsCmd.execute(ListHostsCmd.java:167)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:509)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:416)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:300)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:59)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat
ionFilterChain.java:290)
at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte
rChain.java:206)
at

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve
.java:233)
at

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve
.java:191)
at

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
127)
at

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
102)
at

org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.j
ava:109)
at

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:29
8)
at

org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.ja
va:889)
at

org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.proces
s(Http11NioProtocol.java:721)
at

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.ja
va:2268)
at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
:1110)
at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
a:603)
at java.lang.Thread.run(Thread.java:679)
2012-10-15 09:34:33,597 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-14:null) Ping from 14
2012-10-15 09:34:33,658 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-188:null) Seq 4-857023789: Response Received:
2012-10-15 09:34:33,658 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-188:null) Cleanup succeeded. Details null
2012-10-15 09:34:33,658 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 4-857023789: Received:  { Ans: , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2012-10-15 09:34:33,658 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(StatsCollector-1:null) Cleanup succeeded. Details null
2012-10-15 09:34:33,665 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-70:null) Seq 4-857023790: Executing request
2012-10-15 09:34:34,455 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-70:null) Seq 4-857023790: Response Received:
2012-10-15 09:34:34,456 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-70:null) Cleanup succeeded. Details null
2012-10-15 09:34:34,456 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 4-857023790: Received:  { Ans: , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2012-10-15 09:34:34,456 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(StatsCollector-1:null) Cleanup succeeded. Details null
2012-10-15 09:34:34,461 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-459:null) Seq 4-857023791: Executing request
2012-10-15 09:34:35,387 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-459:null) Seq 4-857023791: Response Received:
2012-10-15 09:34:35,387 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-459:null) Cleanup succeeded. Details null
2012-10-15 09:34:35,388 DEBUG [agent.transport.Request]
(StatsCollector-1:null) Seq 4-857023791: Received:  { Ans: , MgmtId:
130577622632, via: 4, Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2012-10-15 09:34:35,388 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(StatsCollector-1:null) Cleanup succeeded. Details null
2012-10-15 09:34:35,547 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) HostStatsCollector is running...
2012-10-15 09:34:35,556 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-137:null) Seq 4-857023792: Executing request
2

Now, I can't view my hosts under Infrastructure, and can't create new
VMs.  I don't have any local storage in my cloud, so I don't understand
the reference to local storage in the log. When I go to view hosts I get
an empty popup box.  All my storage and hosts are online, so my guess
this is a database error, but I don;t know where to look.

--
Regards,

Nik

Nik Martin
VP Business Development
Nfina Technologies, Inc.
+1.251.243.0043 x1003
Relentless Reliability





--
Regards,

Nik

Nik Martin
VP Business Development
Nfina Technologies, Inc.
+1.251.243.0043 x1003
Relentless Reliability







--
Regards,

Nik

Nik Martin
VP Business Development
Nfina Technologies, Inc.
+1.251.243.0043 x1003
Relentless Reliability

Reply via email to