Well the hypervisior looks fine from a Xenserver prospective but CloudStack won't "reconnect" it. I thought about deleting it in CS and re-adding it, but that bugs me lol
Curtis Old Neustar, Inc. / Neustarlabs / Senior Research Engineer 46000 Center Oak Plaza Sterling, VA 20166 Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis....@neustar.biz<mailto:curtis....@neustar.biz> / www.neustar.biz<http://www.neustar.biz/> ________________________________ The information contained in this e-mail message is intended only for the use of the recipient(s) named above and may contain confidential and/or privileged information. If you are not the intended recipient you have received this e-mail message in error and any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately and delete the original message. From: Ahmad Emneina <aemne...@gmail.com<mailto:aemne...@gmail.com>> Reply-To: "aemne...@gmail.com<mailto:aemne...@gmail.com>" <aemne...@gmail.com<mailto:aemne...@gmail.com>> Date: Tuesday, May 28, 2013 4:56 PM To: "Old, Curtis" <curtis....@neustar.biz<mailto:curtis....@neustar.biz>> Subject: Re: Xenserver hypervisior will not reconnect after reboot are you good to go again, after cleaning up some space? On Tue, May 28, 2013 at 1:24 PM, Old, Curtis <curtis....@neustar.biz<mailto:curtis....@neustar.biz>> wrote: Sorry the log entries are from the xenserver log Curtis Old Neustar, Inc. / Neustarlabs / Senior Research Engineer 46000 Center Oak Plaza Sterling, VA 20166 Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis....@neustar.biz<mailto:curtis....@neustar.biz> / www.neustar.biz<http://www.neustar.biz> <http://www.neustar.biz/> ________________________________________ The information contained in this e-mail message is intended only for the use of the recipient(s) named above and may contain confidential and/or privileged information. If you are not the intended recipient you have received this e-mail message in error and any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately and delete the original message. On 5/28/13 4:20 PM, "Old, Curtis" <curtis....@neustar.biz<mailto:curtis....@neustar.biz>> wrote: >The in innodes did fill, so after I cleaned them up I rebooted and then >the host was singing and dancing again in XenCenter, and primary storage >was mounted. What from the management log, if we can narrow it down? >When I try the forced reconnect only this shows up: > >[20130528T20:17:39.118Z|debug|lab-cloud-9|8622 inet_rpc||http_critical] >Connection terminated >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) connected to >10.31.105.158:443<http://10.31.105.158:443> >[20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23063 >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) connected to >10.31.105.158:443<http://10.31.105.158:443> >[20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23063 >[20130528T20:18:39.124Z|debug|lab-cloud-9|8623 inet_rpc||http_critical] >Connection terminated >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) connected to >10.31.105.158:443<http://10.31.105.158:443> >[20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23068 >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) connected to >10.31.105.158:443<http://10.31.105.158:443> >[20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat >D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23072 > > > > > >Curtis Old >Neustar, Inc. / Neustarlabs / Senior Research Engineer >46000 Center Oak Plaza Sterling, VA 20166 >Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >curtis....@neustar.biz<mailto:curtis....@neustar.biz> >/ www.neustar.biz<http://www.neustar.biz> <http://www.neustar.biz/> >________________________________________ > > >The information contained in this e-mail message is intended only for the >use of the recipient(s) named above and may contain confidential and/or >privileged information. If you are not the intended recipient you have >received this e-mail message in error and any review, dissemination, >distribution, or copying of this message is strictly prohibited. If you >have received this communication in error, please notify us immediately >and delete the original message. > > > > > >On 5/28/13 4:08 PM, "Ahmad Emneina" ><aemne...@gmail.com<mailto:aemne...@gmail.com>> wrote: > >>pretty odd... how does the disk space look on the xenserver host? we'd >>need >>more from the management log, as well as the xensource.log off the >>hypervisor. >> >> >>On Tue, May 28, 2013 at 9:21 AM, Old, Curtis >><curtis....@neustar.biz<mailto:curtis....@neustar.biz>> >>wrote: >> >>> >>> Tried the force reconnect from CS 4.0.0 UI and I get "Command Failed >>>due >>> to internal server Error" any ideas? >>> >>> management-server.log >>> >>> 2013-05-24 10:33:00,974 DEBUG [cloud.async.AsyncJobManagerImpl] >>> (catalina-exec-7:null) submit async job-230, details: AsyncJobVO >>>{id:230, >>> userId: 2, accountId: 2, sessionKey: null, instanceType: Host, >>>instanceId: >>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, cmdOriginator: null, >>> cmdInfo: >>> >>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a653","sessionk >>>e >>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":"136940598077 >>>8 >>>","ctxAccountId":"2","ctxStartEventId":"1514"}, >>> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, >>> processStatus: 0, resultCode: 0, result: null, initMsid: 112939138816, >>> completeMsid: null, lastUpdated: null, lastPolled: null, created: null} >>> 2013-05-24 10:33:00,978 DEBUG [cloud.async.AsyncJobManagerImpl] >>> (Job-Executor-16:job-230) Executing >>>com.cloud.api.commands.ReconnectHostCmd >>> for job-230 >>> 2013-05-24 10:33:00,985 INFO [agent.manager.AgentManagerImpl] >>> (Job-Executor-16:job-230) Unable to disconnect host because it is not >>>in >>> the correct state: host=20; Status=Disconnected >>> 2013-05-24 10:33:00,986 WARN [api.commands.ReconnectHostCmd] >>> (Job-Executor-16:job-230) Exception: >>> com.cloud.api.ServerApiException >>> at >>> >>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java:10 >>>8 >>>) >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>> at >>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) >>> at >>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> at >>> >>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav >>>a >>>:1110) >>> at >>> >>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja >>>v >>>a:603) >>> at java.lang.Thread.run(Thread.java:679) >>> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] >>> (Job-Executor-16:job-230) class com.cloud.api.ServerApiException : null >>> 2013-05-24 10:33:00,986 DEBUG [cloud.async.AsyncJobManagerImpl] >>> (Job-Executor-16:job-230) Complete async job-230, jobStatus: 2, >>>resultCode: >>> 530, result: Error Code: 534 Error text: null >>> 2013-05-24 10:33:05,998 DEBUG [cloud.async.AsyncJobManagerImpl] >>> (catalina-exec-4:null) Async job-230 completed >>> >>> catalina.out >>> >>> INFO [agent.manager.AgentManagerImpl] (Job-Executor-16:job-230) Unable >>>to >>> disconnect host because it is not in the correct state: host=20; >>> Status=Disconnected >>> WARN [api.commands.ReconnectHostCmd] (Job-Executor-16:job-230) >>>Exception: >>> com.cloud.api.ServerApiException >>> at >>> >>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java:10 >>>8 >>>) >>> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>> at >>> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> at >>> >>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav >>>a >>>:1110) >>> at >>> >>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja >>>v >>>a:603) >>> at java.lang.Thread.run(Thread.java:679) >>> WARN [cloud.api.ApiDispatcher] (Job-Executor-16:job-230) class >>> com.cloud.api.ServerApiException : null >>> >>> >>> >
