The in innodes did fill, so after I cleaned them up I rebooted and then the host was singing and dancing again in XenCenter, and primary storage was mounted. What from the management log, if we can narrow it down? When I try the forced reconnect only this shows up:
[20130528T20:17:39.118Z|debug|lab-cloud-9|8622 inet_rpc||http_critical] Connection terminated [20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) connected to 10.31.105.158:443 [20130528T20:17:51.019Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23063 [20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) connected to 10.31.105.158:443 [20130528T20:18:21.089Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23063 [20130528T20:18:39.124Z|debug|lab-cloud-9|8623 inet_rpc||http_critical] Connection terminated [20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) connected to 10.31.105.158:443 [20130528T20:18:51.159Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23068 [20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) connected to 10.31.105.158:443 [20130528T20:19:21.229Z| info|lab-cloud-9|28 heartbeat|Heartbeat D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None s_pid=23072 Curtis Old Neustar, Inc. / Neustarlabs / Senior Research Engineer 46000 Center Oak Plaza Sterling, VA 20166 Office: +1.571.434.5384 Mobile: +1.540.532.2230 / curtis....@neustar.biz / www.neustar.biz <http://www.neustar.biz/> ________________________________________ The information contained in this e-mail message is intended only for the use of the recipient(s) named above and may contain confidential and/or privileged information. If you are not the intended recipient you have received this e-mail message in error and any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately and delete the original message. On 5/28/13 4:08 PM, "Ahmad Emneina" <aemne...@gmail.com> wrote: >pretty odd... how does the disk space look on the xenserver host? we'd >need >more from the management log, as well as the xensource.log off the >hypervisor. > > >On Tue, May 28, 2013 at 9:21 AM, Old, Curtis <curtis....@neustar.biz> >wrote: > >> >> Tried the force reconnect from CS 4.0.0 UI and I get "Command Failed due >> to internal server Error" any ideas? >> >> management-server.log >> >> 2013-05-24 10:33:00,974 DEBUG [cloud.async.AsyncJobManagerImpl] >> (catalina-exec-7:null) submit async job-230, details: AsyncJobVO >>{id:230, >> userId: 2, accountId: 2, sessionKey: null, instanceType: Host, >>instanceId: >> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, cmdOriginator: null, >> cmdInfo: >> >>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a653","sessionke >>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":"1369405980778 >>","ctxAccountId":"2","ctxStartEventId":"1514"}, >> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, >> processStatus: 0, resultCode: 0, result: null, initMsid: 112939138816, >> completeMsid: null, lastUpdated: null, lastPolled: null, created: null} >> 2013-05-24 10:33:00,978 DEBUG [cloud.async.AsyncJobManagerImpl] >> (Job-Executor-16:job-230) Executing >>com.cloud.api.commands.ReconnectHostCmd >> for job-230 >> 2013-05-24 10:33:00,985 INFO [agent.manager.AgentManagerImpl] >> (Job-Executor-16:job-230) Unable to disconnect host because it is not in >> the correct state: host=20; Status=Disconnected >> 2013-05-24 10:33:00,986 WARN [api.commands.ReconnectHostCmd] >> (Job-Executor-16:job-230) Exception: >> com.cloud.api.ServerApiException >> at >> >>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java:108 >>) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >> at >>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) >> at >>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> >>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java >>:1110) >> at >> >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav >>a:603) >> at java.lang.Thread.run(Thread.java:679) >> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] >> (Job-Executor-16:job-230) class com.cloud.api.ServerApiException : null >> 2013-05-24 10:33:00,986 DEBUG [cloud.async.AsyncJobManagerImpl] >> (Job-Executor-16:job-230) Complete async job-230, jobStatus: 2, >>resultCode: >> 530, result: Error Code: 534 Error text: null >> 2013-05-24 10:33:05,998 DEBUG [cloud.async.AsyncJobManagerImpl] >> (catalina-exec-4:null) Async job-230 completed >> >> catalina.out >> >> INFO [agent.manager.AgentManagerImpl] (Job-Executor-16:job-230) Unable >>to >> disconnect host because it is not in the correct state: host=20; >> Status=Disconnected >> WARN [api.commands.ReconnectHostCmd] (Job-Executor-16:job-230) >>Exception: >> com.cloud.api.ServerApiException >> at >> >>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd.java:108 >>) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >> at >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> >>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java >>:1110) >> at >> >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav >>a:603) >> at java.lang.Thread.run(Thread.java:679) >> WARN [cloud.api.ApiDispatcher] (Job-Executor-16:job-230) class >> com.cloud.api.ServerApiException : null >> >> >>
