how about a reboot of the host, then force reconnect.
On Tue, May 28, 2013 at 2:05 PM, Old, Curtis <curtis....@neustar.biz> wrote: > Bummer no goodŠ... > > Curtis Old > Neustar, Inc. / Neustarlabs / Senior Research Engineer > 46000 Center Oak Plaza Sterling, VA 20166 > Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis....@neustar.biz > / www.neustar.biz <http://www.neustar.biz/> > ________________________________________ > > > The information contained in this e-mail message is intended only for the > use of the recipient(s) named above and may contain confidential and/or > privileged information. If you are not the intended recipient you have > received this e-mail message in error and any review, dissemination, > distribution, or copying of this message is strictly prohibited. If you > have received this communication in error, please notify us immediately > and delete the original message. > > > > > > On 5/28/13 5:01 PM, "Ahmad Emneina" <aemne...@gmail.com> wrote: > > >dont do that :D do a toolstack restart on the xenserver host, and a force > >reconnect from cloudstack. see if that makes them play nice. > > > > > >On Tue, May 28, 2013 at 1:58 PM, Old, Curtis <curtis....@neustar.biz> > >wrote: > > > >> Well the hypervisior looks fine from a Xenserver prospective but > >> CloudStack won't "reconnect" it. I thought about deleting it in CS and > >> re-adding it, but that bugs me lol > >> > >> *Curtis Old** > >> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer* > >> > >> *46000 Center Oak Plaza Sterling, VA 20166** > >> **Office:** **+1.571.434.5384** **Mobile: **+1.540.532.2230** **/** ** > >> curtis....@neustar.biz** / **www.neustar.biz** * > >> * > >> ------------------------------ > >> * > >> > >> *The information contained in this e-mail message is intended only for > >> the use of the recipient(s) named above and may contain confidential > >>and/or > >> privileged information. If you are not the intended recipient you have > >> received this e-mail message in error and any review, dissemination, > >> distribution, or copying of this message is strictly prohibited. If you > >> have received this communication in error, please notify us immediately > >>and > >> delete the original message.* > >> > >> From: Ahmad Emneina <aemne...@gmail.com> > >> Reply-To: "aemne...@gmail.com" <aemne...@gmail.com> > >> Date: Tuesday, May 28, 2013 4:56 PM > >> To: "Old, Curtis" <curtis....@neustar.biz> > >> Subject: Re: Xenserver hypervisior will not reconnect after reboot > >> > >> are you good to go again, after cleaning up some space? > >> > >> > >> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis > >><curtis....@neustar.biz>wrote: > >> > >>> Sorry the log entries are from the xenserver log > >>> > >>> Curtis Old > >>> Neustar, Inc. / Neustarlabs / Senior Research Engineer > >>> 46000 Center Oak Plaza Sterling, VA 20166 > >>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / > >>>curtis....@neustar.biz > >>> / www.neustar.biz <http://www.neustar.biz/> > >>> ________________________________________ > >>> > >>> > >>> The information contained in this e-mail message is intended only for > >>>the > >>> use of the recipient(s) named above and may contain confidential and/or > >>> privileged information. If you are not the intended recipient you have > >>> received this e-mail message in error and any review, dissemination, > >>> distribution, or copying of this message is strictly prohibited. If you > >>> have received this communication in error, please notify us immediately > >>> and delete the original message. > >>> > >>> > >>> > >>> > >>> > >>> On 5/28/13 4:20 PM, "Old, Curtis" <curtis....@neustar.biz> wrote: > >>> > >>> >The in innodes did fill, so after I cleaned them up I rebooted and > >>>then > >>> >the host was singing and dancing again in XenCenter, and primary > >>>storage > >>> >was mounted. What from the management log, if we can narrow it down? > >>> >When I try the forced reconnect only this shows up: > >>> > > >>> >[20130528T20:17:39.118Z|debug|lab-cloud-9|8622 > >>>inet_rpc||http_critical] > >>> >Connection terminated > >>> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) connected to > >>> >10.31.105.158:443 > >>> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>s_pid=23063 > >>> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) connected to > >>> >10.31.105.158:443 > >>> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>s_pid=23063 > >>> >[20130528T20:18:39.124Z|debug|lab-cloud-9|8623 > >>>inet_rpc||http_critical] > >>> >Connection terminated > >>> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) connected to > >>> >10.31.105.158:443 > >>> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>s_pid=23068 > >>> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) connected to > >>> >10.31.105.158:443 > >>> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat > >>> >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>s_pid=23072 > >>> > > >>> > > >>> > > >>> > > >>> > > >>> >Curtis Old > >>> >Neustar, Inc. / Neustarlabs / Senior Research Engineer > >>> >46000 Center Oak Plaza Sterling, VA 20166 > >>> >Office: +1.571.434.5384 Mobile: +1.540.532.2230 / > >>>curtis....@neustar.biz > >>> >/ www.neustar.biz <http://www.neustar.biz/> > >>> >________________________________________ > >>> > > >>> > > >>> >The information contained in this e-mail message is intended only for > >>>the > >>> >use of the recipient(s) named above and may contain confidential > >>>and/or > >>> >privileged information. If you are not the intended recipient you have > >>> >received this e-mail message in error and any review, dissemination, > >>> >distribution, or copying of this message is strictly prohibited. If > >>>you > >>> >have received this communication in error, please notify us > >>>immediately > >>> >and delete the original message. > >>> > > >>> > > >>> > > >>> > > >>> > > >>> >On 5/28/13 4:08 PM, "Ahmad Emneina" <aemne...@gmail.com> wrote: > >>> > > >>> >>pretty odd... how does the disk space look on the xenserver host? > >>>we'd > >>> >>need > >>> >>more from the management log, as well as the xensource.log off the > >>> >>hypervisor. > >>> >> > >>> >> > >>> >>On Tue, May 28, 2013 at 9:21 AM, Old, Curtis <curtis....@neustar.biz > > > >>> >>wrote: > >>> >> > >>> >>> > >>> >>> Tried the force reconnect from CS 4.0.0 UI and I get "Command > >>>Failed > >>> >>>due > >>> >>> to internal server Error" any ideas? > >>> >>> > >>> >>> management-server.log > >>> >>> > >>> >>> 2013-05-24 10:33:00,974 DEBUG [cloud.async.AsyncJobManagerImpl] > >>> >>> (catalina-exec-7:null) submit async job-230, details: AsyncJobVO > >>> >>>{id:230, > >>> >>> userId: 2, accountId: 2, sessionKey: null, instanceType: Host, > >>> >>>instanceId: > >>> >>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, cmdOriginator: > >>>null, > >>> >>> cmdInfo: > >>> >>> > >>> > >>> > >>>>>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a653","sessi > >>>>>>onk > >>> >>>e > >>> > >>> > >>>>>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":"136940598 > >>>>>>077 > >>> >>>8 > >>> >>>","ctxAccountId":"2","ctxStartEventId":"1514"}, > >>> >>> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > >>> >>> processStatus: 0, resultCode: 0, result: null, initMsid: > >>>112939138816, > >>> >>> completeMsid: null, lastUpdated: null, lastPolled: null, created: > >>> null} > >>> >>> 2013-05-24 10:33:00,978 DEBUG [cloud.async.AsyncJobManagerImpl] > >>> >>> (Job-Executor-16:job-230) Executing > >>> >>>com.cloud.api.commands.ReconnectHostCmd > >>> >>> for job-230 > >>> >>> 2013-05-24 10:33:00,985 INFO [agent.manager.AgentManagerImpl] > >>> >>> (Job-Executor-16:job-230) Unable to disconnect host because it is > >>>not > >>> >>>in > >>> >>> the correct state: host=20; Status=Disconnected > >>> >>> 2013-05-24 10:33:00,986 WARN [api.commands.ReconnectHostCmd] > >>> >>> (Job-Executor-16:job-230) Exception: > >>> >>> com.cloud.api.ServerApiException > >>> >>> at > >>> >>> > >>> > >>> > >>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java > >>>>>>:10 > >>> >>>8 > >>> >>>) > >>> >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > >>> >>> at > >>> > >>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:43 > >>>>>>2) > >>> >>> at > >>> > >>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47 > >>>>>>1) > >>> >>> at > >>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >>> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > >>> >>> at > >>> >>> > >>> > >>> > >>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. > >>>>>>jav > >>> >>>a > >>> >>>:1110) > >>> >>> at > >>> >>> > >>> > >>> > >>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor > >>>>>>.ja > >>> >>>v > >>> >>>a:603) > >>> >>> at java.lang.Thread.run(Thread.java:679) > >>> >>> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] > >>> >>> (Job-Executor-16:job-230) class com.cloud.api.ServerApiException : > >>> null > >>> >>> 2013-05-24 10:33:00,986 DEBUG [cloud.async.AsyncJobManagerImpl] > >>> >>> (Job-Executor-16:job-230) Complete async job-230, jobStatus: 2, > >>> >>>resultCode: > >>> >>> 530, result: Error Code: 534 Error text: null > >>> >>> 2013-05-24 10:33:05,998 DEBUG [cloud.async.AsyncJobManagerImpl] > >>> >>> (catalina-exec-4:null) Async job-230 completed > >>> >>> > >>> >>> catalina.out > >>> >>> > >>> >>> INFO [agent.manager.AgentManagerImpl] (Job-Executor-16:job-230) > >>> Unable > >>> >>>to > >>> >>> disconnect host because it is not in the correct state: host=20; > >>> >>> Status=Disconnected > >>> >>> WARN [api.commands.ReconnectHostCmd] (Job-Executor-16:job-230) > >>> >>>Exception: > >>> >>> com.cloud.api.ServerApiException > >>> >>> at > >>> >>> > >>> > >>> > >>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java > >>>>>>:10 > >>> >>>8 > >>> >>>) > >>> >>> at > >>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > >>> >>> at > >>> >>> > >>> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) > >>> >>> at > >>> >>> > >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > >>> >>> at > >>> >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >>> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > >>> >>> at > >>> >>> > >>> > >>> > >>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. > >>>>>>jav > >>> >>>a > >>> >>>:1110) > >>> >>> at > >>> >>> > >>> > >>> > >>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor > >>>>>>.ja > >>> >>>v > >>> >>>a:603) > >>> >>> at java.lang.Thread.run(Thread.java:679) > >>> >>> WARN [cloud.api.ApiDispatcher] (Job-Executor-16:job-230) class > >>> >>> com.cloud.api.ServerApiException : null > >>> >>> > >>> >>> > >>> >>> > >>> > > >>> > >>> > >> > >
