I'd def. try Kirk's suggestion first.
On Tue, May 28, 2013 at 3:37 PM, Kirk Kosinski <[email protected]>wrote: > Before hacking the db, I suggest trying one more thing. > > 1. In the CS UI, unmanage the cluster with the affected host. > 2. From the CLI of a host in the XS pool, clear the host tags of the > affected host. > xe host-param-clear param-name=tags uuid=<UUID of affected host> > 3. In the CS UI, manage the cluster with the affected host. > > This usually solves this type of problem (it might take a couple > minutes). Clearing host tags won't affect running VMs (if any). > > Best regards, > Kirk > > On 05/28/2013 03:03 PM, Ahmad Emneina wrote: > > first, I'd query to make sure its the right host, judging from the logs > it > > has an id of 20... execute the following > > backup command from the cli that can access the db server: > > > > mysqldump <credentials if necessary> --databases cloud > cloud.sql > > > > Then start operating... > > > > select * from host where id=20\G (hit the enter key) > > > > if this is infact the misbehaving host, execute > > > > update host set status='Up' where id=20; (again hit enter) > > > > then try to launch a vm on that host (in the deployVirtualMachine api > > command you can/should be able to specify the exact host) > > > > > > On Tue, May 28, 2013 at 2:46 PM, Old, Curtis <[email protected]> > wrote: > > > >> Oh god where do I find that sucker in the DB? Sorry I'M an admin type > >> that isn't a mysql guru or Ji > >> > >> Curtis Old > >> Neustar, Inc. / Neustarlabs / Senior Research Engineer > >> 46000 Center Oak Plaza Sterling, VA 20166 > >> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / > [email protected] > >> / www.neustar.biz <http://www.neustar.biz/> > >> ________________________________________ > >> > >> > >> The information contained in this e-mail message is intended only for > the > >> use of the recipient(s) named above and may contain confidential and/or > >> privileged information. If you are not the intended recipient you have > >> received this e-mail message in error and any review, dissemination, > >> distribution, or copying of this message is strictly prohibited. If you > >> have received this communication in error, please notify us immediately > >> and delete the original message. > >> > >> > >> > >> > >> > >> On 5/28/13 5:43 PM, "Ahmad Emneina" <[email protected]> wrote: > >> > >>> oh this looks like a bug. A potential workaround for it is marking the > >>> host > >>> as Up in the cloudstack db. I'd make a backup first then modify the db > >>> entry. > >>> > >>> > >>> On Tue, May 28, 2013 at 2:36 PM, Old, Curtis <[email protected]> > >>> wrote: > >>> > >>>> Tailed last 3000 lines right after trying a forced reconnect: > >>>> http://paste.cloudstack.org/hTOm/ > >>>> > >>>> Listed as Management log N*, > >>>> > >>>> Author Curtis Old > >>>> > >>>> > >>>> Can we see the latest logs? paste.cloudstack.org > >>>> > >>>> > >>>> On Tue, May 28, 2013 at 2:15 PM, Old, Curtis > >>>> <[email protected]>wrote: > >>>> > >>>>> Tried that already (like 2 times) > >>>>> > >>>>> Curtis Old > >>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer > >>>>> 46000 Center Oak Plaza Sterling, VA 20166 > >>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / > >>>>> [email protected] > >>>>> / www.neustar.biz <http://www.neustar.biz/> > >>>>> ________________________________________ > >>>>> > >>>>> > >>>>> The information contained in this e-mail message is intended only for > >>>>> the > >>>>> use of the recipient(s) named above and may contain confidential > and/or > >>>>> privileged information. If you are not the intended recipient you > have > >>>>> received this e-mail message in error and any review, dissemination, > >>>>> distribution, or copying of this message is strictly prohibited. If > you > >>>>> have received this communication in error, please notify us > immediately > >>>>> and delete the original message. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On 5/28/13 5:12 PM, "Ahmad Emneina" <[email protected]> wrote: > >>>>> > >>>>>> how about a reboot of the host, then force reconnect. > >>>>>> > >>>>>> > >>>>>> On Tue, May 28, 2013 at 2:05 PM, Old, Curtis < > [email protected]> > >>>>>> wrote: > >>>>>> > >>>>>>> Bummer no goodŠ... > >>>>>>> > >>>>>>> Curtis Old > >>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer > >>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 > >>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / > >>>>> [email protected] > >>>>>>> / www.neustar.biz <http://www.neustar.biz/> > >>>>>>> ________________________________________ > >>>>>>> > >>>>>>> > >>>>>>> The information contained in this e-mail message is intended only > >>>>> for > >>>>>>> the > >>>>>>> use of the recipient(s) named above and may contain confidential > >>>>> and/or > >>>>>>> privileged information. If you are not the intended recipient you > >>>>> have > >>>>>>> received this e-mail message in error and any review, > dissemination, > >>>>>>> distribution, or copying of this message is strictly prohibited. If > >>>>> you > >>>>>>> have received this communication in error, please notify us > >>>>> immediately > >>>>>>> and delete the original message. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 5/28/13 5:01 PM, "Ahmad Emneina" <[email protected]> wrote: > >>>>>>> > >>>>>>>> dont do that :D do a toolstack restart on the xenserver host, and > a > >>>>>>> force > >>>>>>>> reconnect from cloudstack. see if that makes them play nice. > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, May 28, 2013 at 1:58 PM, Old, Curtis > >>>>> <[email protected]> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Well the hypervisior looks fine from a Xenserver prospective > >>>>> but > >>>>>>>>> CloudStack won't "reconnect" it. I thought about deleting it in > >>>>> CS > >>>>>>> and > >>>>>>>>> re-adding it, but that bugs me lol > >>>>>>>>> > >>>>>>>>> *Curtis Old** > >>>>>>>>> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer* > >>>>>>>>> > >>>>>>>>> *46000 Center Oak Plaza Sterling, VA 20166** > >>>>>>>>> **Office:** **+1.571.434.5384** **Mobile: **+1.540.532.2230** > >>>>> **/** > >>>>>>> ** > >>>>>>>>> [email protected]** / **www.neustar.biz** * > >>>>>>>>> * > >>>>>>>>> ------------------------------ > >>>>>>>>> * > >>>>>>>>> > >>>>>>>>> *The information contained in this e-mail message is intended > >>>>> only > >>>>>>> for > >>>>>>>>> the use of the recipient(s) named above and may contain > >>>>> confidential > >>>>>>>>> and/or > >>>>>>>>> privileged information. If you are not the intended recipient you > >>>>>>> have > >>>>>>>>> received this e-mail message in error and any review, > >>>>> dissemination, > >>>>>>>>> distribution, or copying of this message is strictly prohibited. > >>>>> If > >>>>>>> you > >>>>>>>>> have received this communication in error, please notify us > >>>>>>> immediately > >>>>>>>>> and > >>>>>>>>> delete the original message.* > >>>>>>>>> > >>>>>>>>> From: Ahmad Emneina <[email protected]> > >>>>>>>>> Reply-To: "[email protected]" <[email protected]> > >>>>>>>>> Date: Tuesday, May 28, 2013 4:56 PM > >>>>>>>>> To: "Old, Curtis" <[email protected]> > >>>>>>>>> Subject: Re: Xenserver hypervisior will not reconnect after > >>>>> reboot > >>>>>>>>> > >>>>>>>>> are you good to go again, after cleaning up some space? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis > >>>>>>>>> <[email protected]>wrote: > >>>>>>>>> > >>>>>>>>>> Sorry the log entries are from the xenserver log > >>>>>>>>>> > >>>>>>>>>> Curtis Old > >>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer > >>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 > >>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / > >>>>>>>>>> [email protected] > >>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> > >>>>>>>>>> ________________________________________ > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> The information contained in this e-mail message is intended > >>>>> only > >>>>>>> for > >>>>>>>>>> the > >>>>>>>>>> use of the recipient(s) named above and may contain confidential > >>>>>>> and/or > >>>>>>>>>> privileged information. If you are not the intended recipient > >>>>> you > >>>>>>> have > >>>>>>>>>> received this e-mail message in error and any review, > >>>>> dissemination, > >>>>>>>>>> distribution, or copying of this message is strictly > >>>>> prohibited. If > >>>>>>> you > >>>>>>>>>> have received this communication in error, please notify us > >>>>>>> immediately > >>>>>>>>>> and delete the original message. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 5/28/13 4:20 PM, "Old, Curtis" <[email protected]> > >>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> The in innodes did fill, so after I cleaned them up I rebooted > >>>>> and > >>>>>>>>>> then > >>>>>>>>>>> the host was singing and dancing again in XenCenter, and > >>>>> primary > >>>>>>>>>> storage > >>>>>>>>>>> was mounted. What from the management log, if we can narrow it > >>>>>>> down? > >>>>>>>>>>> When I try the forced reconnect only this shows up: > >>>>>>>>>>> > >>>>>>>>>>> [20130528T20:17:39.118Z|debug|lab-cloud-9|8622 > >>>>>>>>>> inet_rpc||http_critical] > >>>>>>>>>>> Connection terminated > >>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) > >>>>> connected > >>>>>>> to > >>>>>>>>>>> 10.31.105.158:443 > >>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>>>>>>>> s_pid=23063 > >>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) > >>>>> connected > >>>>>>> to > >>>>>>>>>>> 10.31.105.158:443 > >>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>>>>>>>> s_pid=23063 > >>>>>>>>>>> [20130528T20:18:39.124Z|debug|lab-cloud-9|8623 > >>>>>>>>>> inet_rpc||http_critical] > >>>>>>>>>>> Connection terminated > >>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) > >>>>> connected > >>>>>>> to > >>>>>>>>>>> 10.31.105.158:443 > >>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>>>>>>>> s_pid=23068 > >>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) > >>>>> connected > >>>>>>> to > >>>>>>>>>>> 10.31.105.158:443 > >>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 > >>>>> heartbeat|Heartbeat > >>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None > >>>>>>>>>> s_pid=23072 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Curtis Old > >>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer > >>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 > >>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / > >>>>>>>>>> [email protected] > >>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> > >>>>>>>>>>> ________________________________________ > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> The information contained in this e-mail message is intended > >>>>> only > >>>>>>> for > >>>>>>>>>> the > >>>>>>>>>>> use of the recipient(s) named above and may contain > >>>>> confidential > >>>>>>>>>> and/or > >>>>>>>>>>> privileged information. If you are not the intended recipient > >>>>> you > >>>>>>> have > >>>>>>>>>>> received this e-mail message in error and any review, > >>>>>>> dissemination, > >>>>>>>>>>> distribution, or copying of this message is strictly > >>>>> prohibited. > >>>>> If > >>>>>>>>>> you > >>>>>>>>>>> have received this communication in error, please notify us > >>>>>>>>>> immediately > >>>>>>>>>>> and delete the original message. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On 5/28/13 4:08 PM, "Ahmad Emneina" <[email protected]> > >> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> pretty odd... how does the disk space look on the xenserver > >>>>> host? > >>>>>>>>>> we'd > >>>>>>>>>>>> need > >>>>>>>>>>>> more from the management log, as well as the xensource.log off > >>>>> the > >>>>>>>>>>>> hypervisor. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, May 28, 2013 at 9:21 AM, Old, Curtis > >>>>>>> <[email protected] > >>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Tried the force reconnect from CS 4.0.0 UI and I get > >>>>> "Command > >>>>>>>>>> Failed > >>>>>>>>>>>>> due > >>>>>>>>>>>>> to internal server Error" any ideas? > >>>>>>>>>>>>> > >>>>>>>>>>>>> management-server.log > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2013-05-24 10:33:00,974 DEBUG > >>>>> [cloud.async.AsyncJobManagerImpl] > >>>>>>>>>>>>> (catalina-exec-7:null) submit async job-230, details: > >>>>> AsyncJobVO > >>>>>>>>>>>>> {id:230, > >>>>>>>>>>>>> userId: 2, accountId: 2, sessionKey: null, instanceType: > >>>>> Host, > >>>>>>>>>>>>> instanceId: > >>>>>>>>>>>>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, > >>>>> cmdOriginator: > >>>>>>>>>> null, > >>>>>>>>>>>>> cmdInfo: > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > {"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a653"," > >>>>>>>>>>>>> ses > >>>>>>>>>>>>> si > >>>>>>>>>>>>> onk > >>>>>>>>>>>>> e > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_":"1369 > >>>>>>>>>>>>> 405 > >>>>>>>>>>>>> 98 > >>>>>>>>>>>>> 077 > >>>>>>>>>>>>> 8 > >>>>>>>>>>>>> ","ctxAccountId":"2","ctxStartEventId":"1514"}, > >>>>>>>>>>>>> cmdVersion: 0, callbackType: 0, callbackAddress: null, > >>>>> status: > >>>>>>> 0, > >>>>>>>>>>>>> processStatus: 0, resultCode: 0, result: null, initMsid: > >>>>>>>>>> 112939138816, > >>>>>>>>>>>>> completeMsid: null, lastUpdated: null, lastPolled: null, > >>>>>>> created: > >>>>>>>>>> null} > >>>>>>>>>>>>> 2013-05-24 10:33:00,978 DEBUG > >>>>> [cloud.async.AsyncJobManagerImpl] > >>>>>>>>>>>>> (Job-Executor-16:job-230) Executing > >>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd > >>>>>>>>>>>>> for job-230 > >>>>>>>>>>>>> 2013-05-24 10:33:00,985 INFO > >>>>> [agent.manager.AgentManagerImpl] > >>>>>>>>>>>>> (Job-Executor-16:job-230) Unable to disconnect host because > >>>>> it > >>>>>>> is > >>>>>>>>>> not > >>>>>>>>>>>>> in > >>>>>>>>>>>>> the correct state: host=20; Status=Disconnected > >>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN > >>>>> [api.commands.ReconnectHostCmd] > >>>>>>>>>>>>> (Job-Executor-16:job-230) Exception: > >>>>>>>>>>>>> com.cloud.api.ServerApiException > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd > >>>>>>>>>>>>> .ja > >>>>>>>>>>>>> va > >>>>>>>>>>>>> :10 > >>>>>>>>>>>>> 8 > >>>>>>>>>>>>> ) > >>>>>>>>>>>>> at > >>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > >>>>>>>>>>>>> at > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.ja > >>>>>>>>>>>>> va: > >>>>>>>>>>>>> 43 > >>>>>>>>>>>>> 2) > >>>>>>>>>>>>> at > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.ja > >>>>>>>>>>>>> va: > >>>>>>>>>>>>> 47 > >>>>>>>>>>>>> 1) > >>>>>>>>>>>>> at > >>>>>>> > >>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec > >>>>>>>>>>>>> uto > >>>>>>>>>>>>> r. > >>>>>>>>>>>>> jav > >>>>>>>>>>>>> a > >>>>>>>>>>>>> :1110) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe > >>>>>>>>>>>>> cut > >>>>>>>>>>>>> or > >>>>>>>>>>>>> .ja > >>>>>>>>>>>>> v > >>>>>>>>>>>>> a:603) > >>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) > >>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] > >>>>>>>>>>>>> (Job-Executor-16:job-230) class > >>>>>>> com.cloud.api.ServerApiException : > >>>>>>>>>> null > >>>>>>>>>>>>> 2013-05-24 10:33:00,986 DEBUG > >>>>> [cloud.async.AsyncJobManagerImpl] > >>>>>>>>>>>>> (Job-Executor-16:job-230) Complete async job-230, > >>>>> jobStatus: 2, > >>>>>>>>>>>>> resultCode: > >>>>>>>>>>>>> 530, result: Error Code: 534 Error text: null > >>>>>>>>>>>>> 2013-05-24 10:33:05,998 DEBUG > >>>>> [cloud.async.AsyncJobManagerImpl] > >>>>>>>>>>>>> (catalina-exec-4:null) Async job-230 completed > >>>>>>>>>>>>> > >>>>>>>>>>>>> catalina.out > >>>>>>>>>>>>> > >>>>>>>>>>>>> INFO [agent.manager.AgentManagerImpl] > >>>>> (Job-Executor-16:job-230) > >>>>>>>>>> Unable > >>>>>>>>>>>>> to > >>>>>>>>>>>>> disconnect host because it is not in the correct state: > >>>>> host=20; > >>>>>>>>>>>>> Status=Disconnected > >>>>>>>>>>>>> WARN [api.commands.ReconnectHostCmd] > >>>>> (Job-Executor-16:job-230) > >>>>>>>>>>>>> Exception: > >>>>>>>>>>>>> com.cloud.api.ServerApiException > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHostCmd > >>>>>>>>>>>>> .ja > >>>>>>>>>>>>> va > >>>>>>>>>>>>> :10 > >>>>>>>>>>>>> 8 > >>>>>>>>>>>>> ) > >>>>>>>>>>>>> at > >>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>> > >>>>>>> > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432 > >>>>>>> ) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>> > >>>>>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471 > >>>>>>> ) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > >>>>>>>>>>>>> at > >>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec > >>>>>>>>>>>>> uto > >>>>>>>>>>>>> r. > >>>>>>>>>>>>> jav > >>>>>>>>>>>>> a > >>>>>>>>>>>>> :1110) > >>>>>>>>>>>>> at > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe > >>>>>>>>>>>>> cut > >>>>>>>>>>>>> or > >>>>>>>>>>>>> .ja > >>>>>>>>>>>>> v > >>>>>>>>>> >>>a:603) > >>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) > >>>>>>>>>>>>> WARN [cloud.api.ApiDispatcher] (Job-Executor-16:job-230) > >>>>> class > >>>>>>>>>>>>> com.cloud.api.ServerApiException : null > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >> > >> > > > >
