Well one lab-cloud-9 is in an "Alert" State the other, lab-cloud-10 is "Disconnected", and after a refresh on the UI lab-cloud-9 is back to "Connecting"
Suggestions? On 5/29/13 7:17 AM, "Old, Curtis" <[email protected]> wrote: >Interesting the UUID's in the database where different then the Xenserver >UUID's (there where 2 Xenservers hosts that had inode problems) But then >again the other servers all had different UUID's as well <shrug>. > >Mysql DB: > > id: 21 > name: lab-cloud-10.neustarlab.biz > uuid: 5c65a56a-5a71-44e8-b4ae-89bf81ef4d24 > >And on the Host: > > >uuid=dde1a6d7-8708-451f-a4a8-a9c5be79f57e > >It has been 20 to 25 min and one lab-cloud-9 its state is "Connecting" >lab-cloud-10 is in "Disconnected" and the other 3 hosts (lab-cloud-6, 7, >8) are "Up" > >Wait a bit longer? > > > >On 5/28/13 6:37 PM, "Kirk Kosinski" <[email protected]> wrote: > >>Before hacking the db, I suggest trying one more thing. >> >>1. In the CS UI, unmanage the cluster with the affected host. >>2. From the CLI of a host in the XS pool, clear the host tags of the >>affected host. >>xe host-param-clear param-name=tags uuid=<UUID of affected host> >>3. In the CS UI, manage the cluster with the affected host. >> >>This usually solves this type of problem (it might take a couple >>minutes). Clearing host tags won't affect running VMs (if any). >> >>Best regards, >>Kirk >> >>On 05/28/2013 03:03 PM, Ahmad Emneina wrote: >>> first, I'd query to make sure its the right host, judging from the logs >>>it >>> has an id of 20... execute the following >>> backup command from the cli that can access the db server: >>> >>> mysqldump <credentials if necessary> --databases cloud > cloud.sql >>> >>> Then start operating... >>> >>> select * from host where id=20\G (hit the enter key) >>> >>> if this is infact the misbehaving host, execute >>> >>> update host set status='Up' where id=20; (again hit enter) >>> >>> then try to launch a vm on that host (in the deployVirtualMachine api >>> command you can/should be able to specify the exact host) >>> >>> >>> On Tue, May 28, 2013 at 2:46 PM, Old, Curtis <[email protected]> >>>wrote: >>> >>>> Oh god where do I find that sucker in the DB? Sorry I'M an admin type >>>> that isn't a mysql guru or Ji >>>> >>>> Curtis Old >>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>[email protected] >>>> / www.neustar.biz <http://www.neustar.biz/> >>>> ________________________________________ >>>> >>>> >>>> The information contained in this e-mail message is intended only for >>>>the >>>> use of the recipient(s) named above and may contain confidential >>>>and/or >>>> privileged information. If you are not the intended recipient you have >>>> received this e-mail message in error and any review, dissemination, >>>> distribution, or copying of this message is strictly prohibited. If >>>>you >>>> have received this communication in error, please notify us >>>>immediately >>>> and delete the original message. >>>> >>>> >>>> >>>> >>>> >>>> On 5/28/13 5:43 PM, "Ahmad Emneina" <[email protected]> wrote: >>>> >>>>> oh this looks like a bug. A potential workaround for it is marking >>>>>the >>>>> host >>>>> as Up in the cloudstack db. I'd make a backup first then modify the >>>>>db >>>>> entry. >>>>> >>>>> >>>>> On Tue, May 28, 2013 at 2:36 PM, Old, Curtis <[email protected]> >>>>> wrote: >>>>> >>>>>> Tailed last 3000 lines right after trying a forced reconnect: >>>>>> http://paste.cloudstack.org/hTOm/ >>>>>> >>>>>> Listed as Management log N*, >>>>>> >>>>>> Author Curtis Old >>>>>> >>>>>> >>>>>> Can we see the latest logs? paste.cloudstack.org >>>>>> >>>>>> >>>>>> On Tue, May 28, 2013 at 2:15 PM, Old, Curtis >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> Tried that already (like 2 times) >>>>>>> >>>>>>> Curtis Old >>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>> [email protected] >>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>> ________________________________________ >>>>>>> >>>>>>> >>>>>>> The information contained in this e-mail message is intended only >>>>>>>for >>>>>>> the >>>>>>> use of the recipient(s) named above and may contain confidential >>>>>>>and/or >>>>>>> privileged information. If you are not the intended recipient you >>>>>>>have >>>>>>> received this e-mail message in error and any review, >>>>>>>dissemination, >>>>>>> distribution, or copying of this message is strictly prohibited. If >>>>>>>you >>>>>>> have received this communication in error, please notify us >>>>>>>immediately >>>>>>> and delete the original message. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 5/28/13 5:12 PM, "Ahmad Emneina" <[email protected]> wrote: >>>>>>> >>>>>>>> how about a reboot of the host, then force reconnect. >>>>>>>> >>>>>>>> >>>>>>>> On Tue, May 28, 2013 at 2:05 PM, Old, Curtis >>>>>>>><[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Bummer no goodŠ... >>>>>>>>> >>>>>>>>> Curtis Old >>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>> [email protected] >>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>> ________________________________________ >>>>>>>>> >>>>>>>>> >>>>>>>>> The information contained in this e-mail message is intended only >>>>>>> for >>>>>>>>> the >>>>>>>>> use of the recipient(s) named above and may contain confidential >>>>>>> and/or >>>>>>>>> privileged information. If you are not the intended recipient you >>>>>>> have >>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>dissemination, >>>>>>>>> distribution, or copying of this message is strictly prohibited. >>>>>>>>>If >>>>>>> you >>>>>>>>> have received this communication in error, please notify us >>>>>>> immediately >>>>>>>>> and delete the original message. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 5/28/13 5:01 PM, "Ahmad Emneina" <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> dont do that :D do a toolstack restart on the xenserver host, >>>>>>>>>>and a >>>>>>>>> force >>>>>>>>>> reconnect from cloudstack. see if that makes them play nice. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2013 at 1:58 PM, Old, Curtis >>>>>>> <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Well the hypervisior looks fine from a Xenserver prospective >>>>>>> but >>>>>>>>>>> CloudStack won't "reconnect" it. I thought about deleting it >>>>>>>>>>>in >>>>>>> CS >>>>>>>>> and >>>>>>>>>>> re-adding it, but that bugs me lol >>>>>>>>>>> >>>>>>>>>>> *Curtis Old** >>>>>>>>>>> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer* >>>>>>>>>>> >>>>>>>>>>> *46000 Center Oak Plaza Sterling, VA 20166** >>>>>>>>>>> **Office:** **+1.571.434.5384** **Mobile: **+1.540.532.2230** >>>>>>> **/** >>>>>>>>> ** >>>>>>>>>>> [email protected]** / **www.neustar.biz** * >>>>>>>>>>> * >>>>>>>>>>> ------------------------------ >>>>>>>>>>> * >>>>>>>>>>> >>>>>>>>>>> *The information contained in this e-mail message is intended >>>>>>> only >>>>>>>>> for >>>>>>>>>>> the use of the recipient(s) named above and may contain >>>>>>> confidential >>>>>>>>>>> and/or >>>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>>>>>>you >>>>>>>>> have >>>>>>>>>>> received this e-mail message in error and any review, >>>>>>> dissemination, >>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>>>>>prohibited. >>>>>>> If >>>>>>>>> you >>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>> immediately >>>>>>>>>>> and >>>>>>>>>>> delete the original message.* >>>>>>>>>>> >>>>>>>>>>> From: Ahmad Emneina <[email protected]> >>>>>>>>>>> Reply-To: "[email protected]" <[email protected]> >>>>>>>>>>> Date: Tuesday, May 28, 2013 4:56 PM >>>>>>>>>>> To: "Old, Curtis" <[email protected]> >>>>>>>>>>> Subject: Re: Xenserver hypervisior will not reconnect after >>>>>>> reboot >>>>>>>>>>> >>>>>>>>>>> are you good to go again, after cleaning up some space? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis >>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>> >>>>>>>>>>>> Sorry the log entries are from the xenserver log >>>>>>>>>>>> >>>>>>>>>>>> Curtis Old >>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>>> [email protected] >>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>>> ________________________________________ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The information contained in this e-mail message is intended >>>>>>> only >>>>>>>>> for >>>>>>>>>>>> the >>>>>>>>>>>> use of the recipient(s) named above and may contain >>>>>>>>>>>>confidential >>>>>>>>> and/or >>>>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>> you >>>>>>>>> have >>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>> dissemination, >>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>> prohibited. If >>>>>>>>> you >>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>> immediately >>>>>>>>>>>> and delete the original message. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 5/28/13 4:20 PM, "Old, Curtis" <[email protected]> >>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> The in innodes did fill, so after I cleaned them up I >>>>>>>>>>>>>rebooted >>>>>>> and >>>>>>>>>>>> then >>>>>>>>>>>>> the host was singing and dancing again in XenCenter, and >>>>>>> primary >>>>>>>>>>>> storage >>>>>>>>>>>>> was mounted. What from the management log, if we can narrow >>>>>>>>>>>>>it >>>>>>>>> down? >>>>>>>>>>>>> When I try the forced reconnect only this shows up: >>>>>>>>>>>>> >>>>>>>>>>>>> [20130528T20:17:39.118Z|debug|lab-cloud-9|8622 >>>>>>>>>>>> inet_rpc||http_critical] >>>>>>>>>>>>> Connection terminated >>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) >>>>>>> connected >>>>>>>>> to >>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>> s_pid=23063 >>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) >>>>>>> connected >>>>>>>>> to >>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>> s_pid=23063 >>>>>>>>>>>>> [20130528T20:18:39.124Z|debug|lab-cloud-9|8623 >>>>>>>>>>>> inet_rpc||http_critical] >>>>>>>>>>>>> Connection terminated >>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) >>>>>>> connected >>>>>>>>> to >>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>> s_pid=23068 >>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) >>>>>>> connected >>>>>>>>> to >>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 >>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>> s_pid=23072 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Curtis Old >>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>>> [email protected] >>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The information contained in this e-mail message is intended >>>>>>> only >>>>>>>>> for >>>>>>>>>>>> the >>>>>>>>>>>>> use of the recipient(s) named above and may contain >>>>>>> confidential >>>>>>>>>>>> and/or >>>>>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>> you >>>>>>>>> have >>>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>> dissemination, >>>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>> prohibited. >>>>>>> If >>>>>>>>>>>> you >>>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>>>> immediately >>>>>>>>>>>>> and delete the original message. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 5/28/13 4:08 PM, "Ahmad Emneina" <[email protected]> >>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> pretty odd... how does the disk space look on the xenserver >>>>>>> host? >>>>>>>>>>>> we'd >>>>>>>>>>>>>> need >>>>>>>>>>>>>> more from the management log, as well as the xensource.log >>>>>>>>>>>>>>off >>>>>>> the >>>>>>>>>>>>>> hypervisor. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, May 28, 2013 at 9:21 AM, Old, Curtis >>>>>>>>> <[email protected] >>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tried the force reconnect from CS 4.0.0 UI and I get >>>>>>> "Command >>>>>>>>>>>> Failed >>>>>>>>>>>>>>> due >>>>>>>>>>>>>>> to internal server Error" any ideas? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> management-server.log >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2013-05-24 10:33:00,974 DEBUG >>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>> (catalina-exec-7:null) submit async job-230, details: >>>>>>> AsyncJobVO >>>>>>>>>>>>>>> {id:230, >>>>>>>>>>>>>>> userId: 2, accountId: 2, sessionKey: null, instanceType: >>>>>>> Host, >>>>>>>>>>>>>>> instanceId: >>>>>>>>>>>>>>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, >>>>>>> cmdOriginator: >>>>>>>>>>>> null, >>>>>>>>>>>>>>> cmdInfo: >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a65 >>>>>>>>>>>>>>>3 >>>>>>>>>>>>>>>"," >>>>>>>>>>>>>>> ses >>>>>>>>>>>>>>> si >>>>>>>>>>>>>>> onk >>>>>>>>>>>>>>> e >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":" >>>>>>>>>>>>>>>1 >>>>>>>>>>>>>>>369 >>>>>>>>>>>>>>> 405 >>>>>>>>>>>>>>> 98 >>>>>>>>>>>>>>> 077 >>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>> ","ctxAccountId":"2","ctxStartEventId":"1514"}, >>>>>>>>>>>>>>> cmdVersion: 0, callbackType: 0, callbackAddress: null, >>>>>>> status: >>>>>>>>> 0, >>>>>>>>>>>>>>> processStatus: 0, resultCode: 0, result: null, initMsid: >>>>>>>>>>>> 112939138816, >>>>>>>>>>>>>>> completeMsid: null, lastUpdated: null, lastPolled: null, >>>>>>>>> created: >>>>>>>>>>>> null} >>>>>>>>>>>>>>> 2013-05-24 10:33:00,978 DEBUG >>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>> (Job-Executor-16:job-230) Executing >>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd >>>>>>>>>>>>>>> for job-230 >>>>>>>>>>>>>>> 2013-05-24 10:33:00,985 INFO >>>>>>> [agent.manager.AgentManagerImpl] >>>>>>>>>>>>>>> (Job-Executor-16:job-230) Unable to disconnect host because >>>>>>> it >>>>>>>>> is >>>>>>>>>>>> not >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>> the correct state: host=20; Status=Disconnected >>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN >>>>>>> [api.commands.ReconnectHostCmd] >>>>>>>>>>>>>>> (Job-Executor-16:job-230) Exception: >>>>>>>>>>>>>>> com.cloud.api.ServerApiException >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHos >>>>>>>>>>>>>>>t >>>>>>>>>>>>>>>Cmd >>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>> va >>>>>>>>>>>>>>> :10 >>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>> at >>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>>>>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImp >>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>.ja >>>>>>>>>>>>>>> va: >>>>>>>>>>>>>>> 43 >>>>>>>>>>>>>>> 2) >>>>>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executor >>>>>>>>>>>>>>>s >>>>>>>>>>>>>>>.ja >>>>>>>>>>>>>>> va: >>>>>>>>>>>>>>> 47 >>>>>>>>>>>>>>> 1) >>>>>>>>>>>>>>> at >>>>>>>>> >>>>>>>>>> >>>>>>>>>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334 >>>>>>>>>>) >>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>>>>>>>>>>>>>>E >>>>>>>>>>>>>>>xec >>>>>>>>>>>>>>> uto >>>>>>>>>>>>>>> r. >>>>>>>>>>>>>>> jav >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>> :1110) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>Exe >>>>>>>>>>>>>>> cut >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>> v >>>>>>>>>>>>>>> a:603) >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) >>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] >>>>>>>>>>>>>>> (Job-Executor-16:job-230) class >>>>>>>>> com.cloud.api.ServerApiException : >>>>>>>>>>>> null >>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 DEBUG >>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>> (Job-Executor-16:job-230) Complete async job-230, >>>>>>> jobStatus: 2, >>>>>>>>>>>>>>> resultCode: >>>>>>>>>>>>>>> 530, result: Error Code: 534 Error text: null >>>>>>>>>>>>>>> 2013-05-24 10:33:05,998 DEBUG >>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>> (catalina-exec-4:null) Async job-230 completed >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> catalina.out >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> INFO [agent.manager.AgentManagerImpl] >>>>>>> (Job-Executor-16:job-230) >>>>>>>>>>>> Unable >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> disconnect host because it is not in the correct state: >>>>>>> host=20; >>>>>>>>>>>>>>> Status=Disconnected >>>>>>>>>>>>>>> WARN [api.commands.ReconnectHostCmd] >>>>>>> (Job-Executor-16:job-230) >>>>>>>>>>>>>>> Exception: >>>>>>>>>>>>>>> com.cloud.api.ServerApiException >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHos >>>>>>>>>>>>>>>t >>>>>>>>>>>>>>>Cmd >>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>> va >>>>>>>>>>>>>>> :10 >>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>> at >>>>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java >>>>>>>>>: >>>>>>>>>432 >>>>>>>>> ) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> >>>>>>>>> >>>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java >>>>>>>>>: >>>>>>>>>471 >>>>>>>>> ) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>>>>>>>>>>>>>> at >>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>>>>>>>>>>>>>>E >>>>>>>>>>>>>>>xec >>>>>>>>>>>>>>> uto >>>>>>>>>>>>>>> r. >>>>>>>>>>>>>>> jav >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>> :1110) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>Exe >>>>>>>>>>>>>>> cut >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>> v >>>>>>>>>>>> >>>a:603) >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) >>>>>>>>>>>>>>> WARN [cloud.api.ApiDispatcher] (Job-Executor-16:job-230) >>>>>>> class >>>>>>>>>>>>>>> com.cloud.api.ServerApiException : null >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>>> >>> >> >
