Did you reinstall the hypervisor before reconnecting it back to the pool? Ahmad
On May 30, 2013, at 4:10 AM, "Old, Curtis" <[email protected]> wrote: > Ok both are in an "Alert" state. So I'M kinda at a lost for what to try > next. > > > On 5/30/13 6:55 AM, "Old, Curtis" <[email protected]> wrote: > >> Ok totally weird stuff now. >> >> After trying the previous steps, both Xenserver nodes failed to reconnect >> to the cluster, so I decided to remove and re-add them. I performed the >> following: >> >> 1. Unmanaged the cluster >> 2. Removed the 2 nodes >> 3. Managed the cluster, the 3 remaining nodes went into an Up state. >> 4. Add 1 of the Xenserver nodes, and was going to wait to add the last >> one, if the 1st added successfully. >> >> The result I see both of the 2 nodes trying to connect (one I didn't add >> yet), waiting to see if the complete or fail. >> >> >> >> On 5/29/13 8:39 AM, "Old, Curtis" <[email protected]> wrote: >> >>> Same status after an hour. Should I delete them and re-add them or ?? >>> >>> >>> >>> On 5/29/13 7:30 AM, "Old, Curtis" <[email protected]> wrote: >>> >>>> >>>> >>>> Well one lab-cloud-9 is in an "Alert" State the other, lab-cloud-10 is >>>> "Disconnected", and after a refresh on the UI lab-cloud-9 is back to >>>> "Connecting" >>>> >>>> >>>> Suggestions? >>>> >>>> >>>> >>>> On 5/29/13 7:17 AM, "Old, Curtis" <[email protected]> wrote: >>>> >>>>> Interesting the UUID's in the database where different then the >>>>> Xenserver >>>>> UUID's (there where 2 Xenservers hosts that had inode problems) But >>>>> then >>>>> again the other servers all had different UUID's as well <shrug>. >>>>> >>>>> Mysql DB: >>>>> >>>>> id: 21 >>>>> name: lab-cloud-10.neustarlab.biz >>>>> uuid: 5c65a56a-5a71-44e8-b4ae-89bf81ef4d24 >>>>> >>>>> And on the Host: >>>>> >>>>> >>>>> uuid=dde1a6d7-8708-451f-a4a8-a9c5be79f57e >>>>> >>>>> It has been 20 to 25 min and one lab-cloud-9 its state is "Connecting" >>>>> lab-cloud-10 is in "Disconnected" and the other 3 hosts (lab-cloud-6, >>>>> 7, >>>>> 8) are "Up" >>>>> >>>>> Wait a bit longer? >>>>> >>>>> >>>>> >>>>> On 5/28/13 6:37 PM, "Kirk Kosinski" <[email protected]> wrote: >>>>> >>>>>> Before hacking the db, I suggest trying one more thing. >>>>>> >>>>>> 1. In the CS UI, unmanage the cluster with the affected host. >>>>>> 2. From the CLI of a host in the XS pool, clear the host tags of the >>>>>> affected host. >>>>>> xe host-param-clear param-name=tags uuid=<UUID of affected host> >>>>>> 3. In the CS UI, manage the cluster with the affected host. >>>>>> >>>>>> This usually solves this type of problem (it might take a couple >>>>>> minutes). Clearing host tags won't affect running VMs (if any). >>>>>> >>>>>> Best regards, >>>>>> Kirk >>>>>> >>>>>> On 05/28/2013 03:03 PM, Ahmad Emneina wrote: >>>>>>> first, I'd query to make sure its the right host, judging from the >>>>>>> logs >>>>>>> it >>>>>>> has an id of 20... execute the following >>>>>>> backup command from the cli that can access the db server: >>>>>>> >>>>>>> mysqldump <credentials if necessary> --databases cloud > cloud.sql >>>>>>> >>>>>>> Then start operating... >>>>>>> >>>>>>> select * from host where id=20\G (hit the enter key) >>>>>>> >>>>>>> if this is infact the misbehaving host, execute >>>>>>> >>>>>>> update host set status='Up' where id=20; (again hit enter) >>>>>>> >>>>>>> then try to launch a vm on that host (in the deployVirtualMachine >>>>>>> api >>>>>>> command you can/should be able to specify the exact host) >>>>>>> >>>>>>> >>>>>>> On Tue, May 28, 2013 at 2:46 PM, Old, Curtis >>>>>>> <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Oh god where do I find that sucker in the DB? Sorry I'M an admin >>>>>>>> type >>>>>>>> that isn't a mysql guru or Ji >>>>>>>> >>>>>>>> Curtis Old >>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>> [email protected] >>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>> ________________________________________ >>>>>>>> >>>>>>>> >>>>>>>> The information contained in this e-mail message is intended only >>>>>>>> for >>>>>>>> the >>>>>>>> use of the recipient(s) named above and may contain confidential >>>>>>>> and/or >>>>>>>> privileged information. If you are not the intended recipient you >>>>>>>> have >>>>>>>> received this e-mail message in error and any review, >>>>>>>> dissemination, >>>>>>>> distribution, or copying of this message is strictly prohibited. If >>>>>>>> you >>>>>>>> have received this communication in error, please notify us >>>>>>>> immediately >>>>>>>> and delete the original message. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 5/28/13 5:43 PM, "Ahmad Emneina" <[email protected]> wrote: >>>>>>>> >>>>>>>>> oh this looks like a bug. A potential workaround for it is marking >>>>>>>>> the >>>>>>>>> host >>>>>>>>> as Up in the cloudstack db. I'd make a backup first then modify >>>>>>>>> the >>>>>>>>> db >>>>>>>>> entry. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, May 28, 2013 at 2:36 PM, Old, Curtis >>>>>>>>> <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Tailed last 3000 lines right after trying a forced reconnect: >>>>>>>>>> http://paste.cloudstack.org/hTOm/ >>>>>>>>>> >>>>>>>>>> Listed as Management log N*, >>>>>>>>>> >>>>>>>>>> Author Curtis Old >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can we see the latest logs? paste.cloudstack.org >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, May 28, 2013 at 2:15 PM, Old, Curtis >>>>>>>>>> <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> Tried that already (like 2 times) >>>>>>>>>>> >>>>>>>>>>> Curtis Old >>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>> [email protected] >>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>> ________________________________________ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The information contained in this e-mail message is intended >>>>>>>>>>> only >>>>>>>>>>> for >>>>>>>>>>> the >>>>>>>>>>> use of the recipient(s) named above and may contain confidential >>>>>>>>>>> and/or >>>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>>>>>> you >>>>>>>>>>> have >>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>>> dissemination, >>>>>>>>>>> distribution, or copying of this message is strictly prohibited. >>>>>>>>>>> If >>>>>>>>>>> you >>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>>> immediately >>>>>>>>>>> and delete the original message. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 5/28/13 5:12 PM, "Ahmad Emneina" <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> how about a reboot of the host, then force reconnect. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2013 at 2:05 PM, Old, Curtis >>>>>>>>>>>> <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Bummer no goodŠ... >>>>>>>>>>>>> >>>>>>>>>>>>> Curtis Old >>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>> [email protected] >>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The information contained in this e-mail message is intended >>>>>>>>>>>>> only >>>>>>>>>>> for >>>>>>>>>>>>> the >>>>>>>>>>>>> use of the recipient(s) named above and may contain >>>>>>>>>>>>> confidential >>>>>>>>>>> and/or >>>>>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>>>>>>>> you >>>>>>>>>>> have >>>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>>>>> dissemination, >>>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>>>>>>> prohibited. >>>>>>>>>>>>> If >>>>>>>>>>> you >>>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>>> immediately >>>>>>>>>>>>> and delete the original message. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 5/28/13 5:01 PM, "Ahmad Emneina" <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> dont do that :D do a toolstack restart on the xenserver host, >>>>>>>>>>>>>> and a >>>>>>>>>>>>> force >>>>>>>>>>>>>> reconnect from cloudstack. see if that makes them play nice. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, May 28, 2013 at 1:58 PM, Old, Curtis >>>>>>>>>>> <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Well the hypervisior looks fine from a Xenserver >>>>>>>>>>>>>>> prospective >>>>>>>>>>> but >>>>>>>>>>>>>>> CloudStack won't "reconnect" it. I thought about deleting >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>> in >>>>>>>>>>> CS >>>>>>>>>>>>> and >>>>>>>>>>>>>>> re-adding it, but that bugs me lol >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Curtis Old** >>>>>>>>>>>>>>> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *46000 Center Oak Plaza Sterling, VA 20166** >>>>>>>>>>>>>>> **Office:** **+1.571.434.5384** **Mobile: >>>>>>>>>>>>>>> **+1.540.532.2230** >>>>>>>>>>> **/** >>>>>>>>>>>>> ** >>>>>>>>>>>>>>> [email protected]** / **www.neustar.biz** * >>>>>>>>>>>>>>> * >>>>>>>>>>>>>>> ------------------------------ >>>>>>>>>>>>>>> * >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *The information contained in this e-mail message is >>>>>>>>>>>>>>> intended >>>>>>>>>>> only >>>>>>>>>>>>> for >>>>>>>>>>>>>>> the use of the recipient(s) named above and may contain >>>>>>>>>>> confidential >>>>>>>>>>>>>>> and/or >>>>>>>>>>>>>>> privileged information. If you are not the intended >>>>>>>>>>>>>>> recipient >>>>>>>>>>>>>>> you >>>>>>>>>>>>> have >>>>>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>>> dissemination, >>>>>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>>>>>>>>> prohibited. >>>>>>>>>>> If >>>>>>>>>>>>> you >>>>>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>>>>> immediately >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> delete the original message.* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Ahmad Emneina <[email protected]> >>>>>>>>>>>>>>> Reply-To: "[email protected]" <[email protected]> >>>>>>>>>>>>>>> Date: Tuesday, May 28, 2013 4:56 PM >>>>>>>>>>>>>>> To: "Old, Curtis" <[email protected]> >>>>>>>>>>>>>>> Subject: Re: Xenserver hypervisior will not reconnect after >>>>>>>>>>> reboot >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> are you good to go again, after cleaning up some space? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis >>>>>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sorry the log entries are from the xenserver log >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Curtis Old >>>>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The information contained in this e-mail message is >>>>>>>>>>>>>>>> intended >>>>>>>>>>> only >>>>>>>>>>>>> for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> use of the recipient(s) named above and may contain >>>>>>>>>>>>>>>> confidential >>>>>>>>>>>>> and/or >>>>>>>>>>>>>>>> privileged information. If you are not the intended >>>>>>>>>>>>>>>> recipient >>>>>>>>>>> you >>>>>>>>>>>>> have >>>>>>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>>> dissemination, >>>>>>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>>>>> prohibited. If >>>>>>>>>>>>> you >>>>>>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>>>>> immediately >>>>>>>>>>>>>>>> and delete the original message. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 5/28/13 4:20 PM, "Old, Curtis" <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The in innodes did fill, so after I cleaned them up I >>>>>>>>>>>>>>>>> rebooted >>>>>>>>>>> and >>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>> the host was singing and dancing again in XenCenter, and >>>>>>>>>>> primary >>>>>>>>>>>>>>>> storage >>>>>>>>>>>>>>>>> was mounted. What from the management log, if we can >>>>>>>>>>>>>>>>> narrow >>>>>>>>>>>>>>>>> it >>>>>>>>>>>>> down? >>>>>>>>>>>>>>>>> When I try the forced reconnect only this shows up: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [20130528T20:17:39.118Z|debug|lab-cloud-9|8622 >>>>>>>>>>>>>>>> inet_rpc||http_critical] >>>>>>>>>>>>>>>>> Connection terminated >>>>>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) >>>>>>>>>>> connected >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid >>>>>>>>>>>>>>>>> task_opt=None >>>>>>>>>>>>>>>> s_pid=23063 >>>>>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) >>>>>>>>>>> connected >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid >>>>>>>>>>>>>>>>> task_opt=None >>>>>>>>>>>>>>>> s_pid=23063 >>>>>>>>>>>>>>>>> [20130528T20:18:39.124Z|debug|lab-cloud-9|8623 >>>>>>>>>>>>>>>> inet_rpc||http_critical] >>>>>>>>>>>>>>>>> Connection terminated >>>>>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) >>>>>>>>>>> connected >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid >>>>>>>>>>>>>>>>> task_opt=None >>>>>>>>>>>>>>>> s_pid=23068 >>>>>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) >>>>>>>>>>> connected >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 >>>>>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid >>>>>>>>>>>>>>>>> task_opt=None >>>>>>>>>>>>>>>> s_pid=23072 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Curtis Old >>>>>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The information contained in this e-mail message is >>>>>>>>>>>>>>>>> intended >>>>>>>>>>> only >>>>>>>>>>>>> for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> use of the recipient(s) named above and may contain >>>>>>>>>>> confidential >>>>>>>>>>>>>>>> and/or >>>>>>>>>>>>>>>>> privileged information. If you are not the intended >>>>>>>>>>>>>>>>> recipient >>>>>>>>>>> you >>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>>>>> dissemination, >>>>>>>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>>>>> prohibited. >>>>>>>>>>> If >>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>> have received this communication in error, please notify >>>>>>>>>>>>>>>>> us >>>>>>>>>>>>>>>> immediately >>>>>>>>>>>>>>>>> and delete the original message. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 5/28/13 4:08 PM, "Ahmad Emneina" <[email protected]> >>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> pretty odd... how does the disk space look on the >>>>>>>>>>>>>>>>>> xenserver >>>>>>>>>>> host? >>>>>>>>>>>>>>>> we'd >>>>>>>>>>>>>>>>>> need >>>>>>>>>>>>>>>>>> more from the management log, as well as the >>>>>>>>>>>>>>>>>> xensource.log >>>>>>>>>>>>>>>>>> off >>>>>>>>>>> the >>>>>>>>>>>>>>>>>> hypervisor. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, May 28, 2013 at 9:21 AM, Old, Curtis >>>>>>>>>>>>> <[email protected] >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Tried the force reconnect from CS 4.0.0 UI and I get >>>>>>>>>>> "Command >>>>>>>>>>>>>>>> Failed >>>>>>>>>>>>>>>>>>> due >>>>>>>>>>>>>>>>>>> to internal server Error" any ideas? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> management-server.log >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,974 DEBUG >>>>>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>>>>> (catalina-exec-7:null) submit async job-230, details: >>>>>>>>>>> AsyncJobVO >>>>>>>>>>>>>>>>>>> {id:230, >>>>>>>>>>>>>>>>>>> userId: 2, accountId: 2, sessionKey: null, instanceType: >>>>>>>>>>> Host, >>>>>>>>>>>>>>>>>>> instanceId: >>>>>>>>>>>>>>>>>>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, >>>>>>>>>>> cmdOriginator: >>>>>>>>>>>>>>>> null, >>>>>>>>>>>>>>>>>>> cmdInfo: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> {"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2 >>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> 6 >>>>>>>>>>>>>>>>>>> 5 >>>>>>>>>>>>>>>>>>> 3 >>>>>>>>>>>>>>>>>>> "," >>>>>>>>>>>>>>>>>>> ses >>>>>>>>>>>>>>>>>>> si >>>>>>>>>>>>>>>>>>> onk >>>>>>>>>>>>>>>>>>> e >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_ >>>>>>>>>>>>>>>>>>> " >>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>>> " >>>>>>>>>>>>>>>>>>> 1 >>>>>>>>>>>>>>>>>>> 369 >>>>>>>>>>>>>>>>>>> 405 >>>>>>>>>>>>>>>>>>> 98 >>>>>>>>>>>>>>>>>>> 077 >>>>>>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>>>>>> ","ctxAccountId":"2","ctxStartEventId":"1514"}, >>>>>>>>>>>>>>>>>>> cmdVersion: 0, callbackType: 0, callbackAddress: null, >>>>>>>>>>> status: >>>>>>>>>>>>> 0, >>>>>>>>>>>>>>>>>>> processStatus: 0, resultCode: 0, result: null, initMsid: >>>>>>>>>>>>>>>> 112939138816, >>>>>>>>>>>>>>>>>>> completeMsid: null, lastUpdated: null, lastPolled: null, >>>>>>>>>>>>> created: >>>>>>>>>>>>>>>> null} >>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,978 DEBUG >>>>>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Executing >>>>>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd >>>>>>>>>>>>>>>>>>> for job-230 >>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,985 INFO >>>>>>>>>>> [agent.manager.AgentManagerImpl] >>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Unable to disconnect host >>>>>>>>>>>>>>>>>>> because >>>>>>>>>>> it >>>>>>>>>>>>> is >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> the correct state: host=20; Status=Disconnected >>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN >>>>>>>>>>> [api.commands.ReconnectHostCmd] >>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Exception: >>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd.execute(Reconnect >>>>>>>>>>>>>>>>>>> H >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> s >>>>>>>>>>>>>>>>>>> t >>>>>>>>>>>>>>>>>>> Cmd >>>>>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>>>>> va >>>>>>>>>>>>>>>>>>> :10 >>>>>>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManager >>>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>> m >>>>>>>>>>>>>>>>>>> p >>>>>>>>>>>>>>>>>>> l >>>>>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>>>>> va: >>>>>>>>>>>>>>>>>>> 43 >>>>>>>>>>>>>>>>>>> 2) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Execu >>>>>>>>>>>>>>>>>>> t >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> r >>>>>>>>>>>>>>>>>>> s >>>>>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>>>>> va: >>>>>>>>>>>>>>>>>>> 47 >>>>>>>>>>>>>>>>>>> 1) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java: >>>>>>>>>>>>>> 3 >>>>>>>>>>>>>> 3 >>>>>>>>>>>>>> 4 >>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadP >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> l >>>>>>>>>>>>>>>>>>> E >>>>>>>>>>>>>>>>>>> xec >>>>>>>>>>>>>>>>>>> uto >>>>>>>>>>>>>>>>>>> r. >>>>>>>>>>>>>>>>>>> jav >>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> :1110) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Thread >>>>>>>>>>>>>>>>>>> P >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> l >>>>>>>>>>>>>>>>>>> Exe >>>>>>>>>>>>>>>>>>> cut >>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>>>>> v >>>>>>>>>>>>>>>>>>> a:603) >>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) >>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] >>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) class >>>>>>>>>>>>> com.cloud.api.ServerApiException : >>>>>>>>>>>>>>>> null >>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 DEBUG >>>>>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Complete async job-230, >>>>>>>>>>> jobStatus: 2, >>>>>>>>>>>>>>>>>>> resultCode: >>>>>>>>>>>>>>>>>>> 530, result: Error Code: 534 Error text: null >>>>>>>>>>>>>>>>>>> 2013-05-24 10:33:05,998 DEBUG >>>>>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>>>>> (catalina-exec-4:null) Async job-230 completed >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> catalina.out >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> INFO [agent.manager.AgentManagerImpl] >>>>>>>>>>> (Job-Executor-16:job-230) >>>>>>>>>>>>>>>> Unable >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> disconnect host because it is not in the correct state: >>>>>>>>>>> host=20; >>>>>>>>>>>>>>>>>>> Status=Disconnected >>>>>>>>>>>>>>>>>>> WARN [api.commands.ReconnectHostCmd] >>>>>>>>>>> (Job-Executor-16:job-230) >>>>>>>>>>>>>>>>>>> Exception: >>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd.execute(Reconnect >>>>>>>>>>>>>>>>>>> H >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> s >>>>>>>>>>>>>>>>>>> t >>>>>>>>>>>>>>>>>>> Cmd >>>>>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>>>>> va >>>>>>>>>>>>>>>>>>> :10 >>>>>>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.j >>>>>>>>>>>>> a >>>>>>>>>>>>> v >>>>>>>>>>>>> a >>>>>>>>>>>>> : >>>>>>>>>>>>> 432 >>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.j >>>>>>>>>>>>> a >>>>>>>>>>>>> v >>>>>>>>>>>>> a >>>>>>>>>>>>> : >>>>>>>>>>>>> 471 >>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>> >>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:3 >>>>>>>>>>>>> 3 >>>>>>>>>>>>> 4 >>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadP >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> l >>>>>>>>>>>>>>>>>>> E >>>>>>>>>>>>>>>>>>> xec >>>>>>>>>>>>>>>>>>> uto >>>>>>>>>>>>>>>>>>> r. >>>>>>>>>>>>>>>>>>> jav >>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> :1110) >>>>>>>>>>>>>>>>>>> at >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Thread >>>>>>>>>>>>>>>>>>> P >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> o >>>>>>>>>>>>>>>>>>> l >>>>>>>>>>>>>>>>>>> Exe >>>>>>>>>>>>>>>>>>> cut >>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>>>>> v >>>>>>>>>>>>>>>>>>> a:603) >>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) >>>>>>>>>>>>>>>>>>> WARN [cloud.api.ApiDispatcher] >>>>>>>>>>>>>>>>>>> (Job-Executor-16:job-230) >>>>>>>>>>> class >>>>>>>>>>>>>>>>>>> com.cloud.api.ServerApiException : null >
