dont do that :D do a toolstack restart on the xenserver host, and a force reconnect from cloudstack. see if that makes them play nice.
On Tue, May 28, 2013 at 1:58 PM, Old, Curtis <curtis....@neustar.biz> wrote: > Well the hypervisior looks fine from a Xenserver prospective but > CloudStack won't "reconnect" it. I thought about deleting it in CS and > re-adding it, but that bugs me lol > > *Curtis Old** > **Neustar, Inc. / Neustarlabs / **Senior Research Engineer* > > *46000 Center Oak Plaza Sterling, VA 20166** > **Office:** **+1.571.434.5384** **Mobile: **+1.540.532.2230** **/** ** > curtis....@neustar.biz** / **www.neustar.biz** * > * > ------------------------------ > * > > *The information contained in this e-mail message is intended only for > the use of the recipient(s) named above and may contain confidential and/or > privileged information. If you are not the intended recipient you have > received this e-mail message in error and any review, dissemination, > distribution, or copying of this message is strictly prohibited. If you > have received this communication in error, please notify us immediately and > delete the original message.* > > From: Ahmad Emneina <aemne...@gmail.com> > Reply-To: "aemne...@gmail.com" <aemne...@gmail.com> > Date: Tuesday, May 28, 2013 4:56 PM > To: "Old, Curtis" <curtis....@neustar.biz> > Subject: Re: Xenserver hypervisior will not reconnect after reboot > > are you good to go again, after cleaning up some space? > > > On Tue, May 28, 2013 at 1:24 PM, Old, Curtis <curtis....@neustar.biz>wrote: > >> Sorry the log entries are from the xenserver log >> >> Curtis Old >> Neustar, Inc. / Neustarlabs / Senior Research Engineer >> 46000 Center Oak Plaza Sterling, VA 20166 >> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis....@neustar.biz >> / www.neustar.biz <http://www.neustar.biz/> >> ________________________________________ >> >> >> The information contained in this e-mail message is intended only for the >> use of the recipient(s) named above and may contain confidential and/or >> privileged information. If you are not the intended recipient you have >> received this e-mail message in error and any review, dissemination, >> distribution, or copying of this message is strictly prohibited. If you >> have received this communication in error, please notify us immediately >> and delete the original message. >> >> >> >> >> >> On 5/28/13 4:20 PM, "Old, Curtis" <curtis....@neustar.biz> wrote: >> >> >The in innodes did fill, so after I cleaned them up I rebooted and then >> >the host was singing and dancing again in XenCenter, and primary storage >> >was mounted. What from the management log, if we can narrow it down? >> >When I try the forced reconnect only this shows up: >> > >> >[20130528T20:17:39.118Z|debug|lab-cloud-9|8622 inet_rpc||http_critical] >> >Connection terminated >> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) connected to >> >10.31.105.158:443 >> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23063 >> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) connected to >> >10.31.105.158:443 >> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23063 >> >[20130528T20:18:39.124Z|debug|lab-cloud-9|8623 inet_rpc||http_critical] >> >Connection terminated >> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) connected to >> >10.31.105.158:443 >> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23068 >> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) connected to >> >10.31.105.158:443 >> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat >> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23072 >> > >> > >> > >> > >> > >> >Curtis Old >> >Neustar, Inc. / Neustarlabs / Senior Research Engineer >> >46000 Center Oak Plaza Sterling, VA 20166 >> >Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis....@neustar.biz >> >/ www.neustar.biz <http://www.neustar.biz/> >> >________________________________________ >> > >> > >> >The information contained in this e-mail message is intended only for the >> >use of the recipient(s) named above and may contain confidential and/or >> >privileged information. If you are not the intended recipient you have >> >received this e-mail message in error and any review, dissemination, >> >distribution, or copying of this message is strictly prohibited. If you >> >have received this communication in error, please notify us immediately >> >and delete the original message. >> > >> > >> > >> > >> > >> >On 5/28/13 4:08 PM, "Ahmad Emneina" <aemne...@gmail.com> wrote: >> > >> >>pretty odd... how does the disk space look on the xenserver host? we'd >> >>need >> >>more from the management log, as well as the xensource.log off the >> >>hypervisor. >> >> >> >> >> >>On Tue, May 28, 2013 at 9:21 AM, Old, Curtis <curtis....@neustar.biz> >> >>wrote: >> >> >> >>> >> >>> Tried the force reconnect from CS 4.0.0 UI and I get "Command Failed >> >>>due >> >>> to internal server Error" any ideas? >> >>> >> >>> management-server.log >> >>> >> >>> 2013-05-24 10:33:00,974 DEBUG [cloud.async.AsyncJobManagerImpl] >> >>> (catalina-exec-7:null) submit async job-230, details: AsyncJobVO >> >>>{id:230, >> >>> userId: 2, accountId: 2, sessionKey: null, instanceType: Host, >> >>>instanceId: >> >>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, cmdOriginator: null, >> >>> cmdInfo: >> >>> >> >> >>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a653","sessionk >> >>>e >> >> >>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":"136940598077 >> >>>8 >> >>>","ctxAccountId":"2","ctxStartEventId":"1514"}, >> >>> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, >> >>> processStatus: 0, resultCode: 0, result: null, initMsid: 112939138816, >> >>> completeMsid: null, lastUpdated: null, lastPolled: null, created: >> null} >> >>> 2013-05-24 10:33:00,978 DEBUG [cloud.async.AsyncJobManagerImpl] >> >>> (Job-Executor-16:job-230) Executing >> >>>com.cloud.api.commands.ReconnectHostCmd >> >>> for job-230 >> >>> 2013-05-24 10:33:00,985 INFO [agent.manager.AgentManagerImpl] >> >>> (Job-Executor-16:job-230) Unable to disconnect host because it is not >> >>>in >> >>> the correct state: host=20; Status=Disconnected >> >>> 2013-05-24 10:33:00,986 WARN [api.commands.ReconnectHostCmd] >> >>> (Job-Executor-16:job-230) Exception: >> >>> com.cloud.api.ServerApiException >> >>> at >> >>> >> >> >>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java:10 >> >>>8 >> >>>) >> >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >> >>> at >> >>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) >> >>> at >> >>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >>> at >> >>> >> >> >>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav >> >>>a >> >>>:1110) >> >>> at >> >>> >> >> >>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja >> >>>v >> >>>a:603) >> >>> at java.lang.Thread.run(Thread.java:679) >> >>> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] >> >>> (Job-Executor-16:job-230) class com.cloud.api.ServerApiException : >> null >> >>> 2013-05-24 10:33:00,986 DEBUG [cloud.async.AsyncJobManagerImpl] >> >>> (Job-Executor-16:job-230) Complete async job-230, jobStatus: 2, >> >>>resultCode: >> >>> 530, result: Error Code: 534 Error text: null >> >>> 2013-05-24 10:33:05,998 DEBUG [cloud.async.AsyncJobManagerImpl] >> >>> (catalina-exec-4:null) Async job-230 completed >> >>> >> >>> catalina.out >> >>> >> >>> INFO [agent.manager.AgentManagerImpl] (Job-Executor-16:job-230) >> Unable >> >>>to >> >>> disconnect host because it is not in the correct state: host=20; >> >>> Status=Disconnected >> >>> WARN [api.commands.ReconnectHostCmd] (Job-Executor-16:job-230) >> >>>Exception: >> >>> com.cloud.api.ServerApiException >> >>> at >> >>> >> >> >>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java:10 >> >>>8 >> >>>) >> >>> at >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >> >>> at >> >>> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) >> >>> at >> >>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >>> at >> >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >>> at >> >>> >> >> >>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav >> >>>a >> >>>:1110) >> >>> at >> >>> >> >> >>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja >> >>>v >> >>>a:603) >> >>> at java.lang.Thread.run(Thread.java:679) >> >>> WARN [cloud.api.ApiDispatcher] (Job-Executor-16:job-230) class >> >>> com.cloud.api.ServerApiException : null >> >>> >> >>> >> >>> >> > >> >> >
